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ABSTRACT 

This set of 82 summaries is based on papers written 
between 1963 and 1982 concerning the Test of English as a Foreign 
Language (TOEFL). TOEFL is currently required by more than 2,500 
colleges and universities in the United States and Canada to 
determine the English proficiency of applicants whose native language 
is not English. It is also required by various agencies and boards 
that accredit and license professionals trained abroad. While a few 
descriptive papers are included to provide historical background, the 
summarized papers are primarily research reports. Studies focussing 
on other instruments which provide data that relate performanre on 
those instruments to TOEFL are also included. The summaries are 
indexed by nine major categories: (1) TOEFL' s relationship to 
similar, standardized objective English language proficiency tests; 
(2) TOEFL' s relationship to otht»r measures ol English language 
proficiency; (3) T0::FL's relationship to tests of intelligence, 
academic aptitude, and achievement; (4) TOEFL ' s relationship to later 
academic performance; (5) simple relation of TOEFL to student 
characteristics; (6) complex relations involving student 
characteri St ics (7) statistical analysis involving TOEFL; (8) 
miscellaneous research issu2s; and (9) general descriptive papers. 
Papers are also indexed in 33 subcategories by types of TOEFL results 
reported. (BS) 
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The Test of English as a Foreign Language (TOEFL) was developed in 1^3 by 
a National Council on the Testing of English as a Foreign Language, which 
was fcnned through the cooperative effort of over thirty organizations, public 
and private, that were concerned with testing the English proficiency of non- 
native speakers of the language applying for admission to institutions in the 
United States, In 1965. Educational Testing Service (ETS) and the College 
Board assumed joint responsibility for the program and in 1973 a cooperative 
arrangement for the operation of the program was entered into by ETS. the 
College Board, and the Graduate Record Examinations (GRE) Board. The 
membership of the College Board is composed of schools, colleges, school 
systems, and educational associations; ORE Board members are associated 
with graduate education. 

ETS administers the TOEFL program under the general direction of a Policy 
Council that was established by, and is affiliated with, the sponsoring organi- 
zations. Members of the Policy Council represent the Collego Board and the 
GRE Board and such institutions and agencies as graduate schools of 
business, junior and community colleges, nonprofit educational exchange 
agencies, and agencies of the United States government. 



A continuing program of research related to TOEFL is carried out under the 
direction of the TOEFL Research Committee, Its six members include repre- 
sentatives of the Policy Council, the TOEFL Committee of Examiners, and 
distinguished English-as-a-second-Ianguage specialists from the academic 
community. Currently the committee meets twice yearly to review and ap- 
prove proposals for test-related research and to set guidelines for the entire 
scope of the TOEFL research program. Members of the Research Committee 
serve three year terms at the invitation of the Policy Council; the chair of the 
committee serves on the Policy Council 

Because the studies are specific to the test and the testing program, most of 
the actual research is conducted by ETS staff rather than by outside re 
searchers. However, many projects require the cooperation of other institu- 
tions, particularly those with programs in the teaching of English as a foreign 
or second language. Representatives of such programs who are interested in 
participating in or conduct'ng TOEFl /^jlated research are invited to contact 
the TOEFL program office. Local ''Coearch may sometimes require access to 
TOEFL data, in such cases, the program may provide this data following 
approval by the Research Committee. All TOEFL research projects must 
u "''?rgo appropriate ETS review to ascertain that the confidentiality of data 
will be protected. 

Current (H>82'83) members of the TOEFL Research Committee include the fot- 
lowing: 

Richard Tucker (chair) Center for Applied Linguistics 

Ali.^on d' Angle/an-Chatillon University of Montreal 
Louis A. Arena University of Delaware 

H, Douglas Brown San Francisco State University 

Frances B, Hinofotis University of California at Los Angeles 

Henr*' F. Holtzclaw, Jr. University of Nebraska 
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Abstract 



This set of 82 suooarles Is based on papers written between 1963 and 
1982 concerning TOEFL. The papers summarized consist ptimarily of reports 
of research involving l^EFL. Also inclisded are a few disspriptive papers, 
which are summarised here to provide a perspective on the history and 
development of the test. The methods and criteria by which papers were 
identified for inclusion, as well as a scheme for classification of the 
papers, are presented in the Introduction. 



INTRODUCTION 



Background 



The Test of English as a Foreign Language (TOEFL) is currently 
required by more than 2,500 colleges and universities in the United States 
and Canada to determine the English proficiency of applicants where native 
language is not English. It is also required by various agencies and 
boards in the United States and Canada concerned with the accreditation 
and licensure of professionals trained abroad. Because of its prominence 
in the field of second language testing, TOE' % has been the subject of 
continual research since its development in 1963. 

Many different issues have been addressed in this research. Studies 
correlating TOEFL with other standardized tests of English language 
proficiency have addressed the issue of TOEFL' s concurrent validity. 
Also relevant to concurrent validity are studies correlating TOEFL with 
performance on cloze tests, dictation measures, oral interviews, writing 
samples, instructors' ratings, and other direct and indirect indices of 
English language proficiency. TOEFL' s role as a predictor of academic 
performance has been examined through studies correlating TOEFL with 
students' grade'polnt averages. Still other correlational studies have 
examined the test's relation to measures of aptitude, intelligence, and 
achievement. 

Another general group of studies has been concerned with the perform- 
ance of different groups of examinees on TOEFL. Within this category are 
studies of various language or culture groups, studies of special popula- 
tions, and comparisons of native English speakers' performance on TOEFL 
with that of nonnative speakers. Still other studies have been concerned 
with statistical analysis of TOEFL or have addressed such issues as 
effects of instruction on TOEFL, characteristics of TOEFL candidates, 
effects of living environment, and others. 

In light of the large amount of research that has now been conducted 
involving TOZrL, the authors, with support provided by the TOEFL Research 
Committee, have collected Information regarding this reseaich. The 
present document consists of summaries of studies involving TOEFL that 
have been conducted from its initial development In 1963 through 1982. 
The studies summarized here include not only those that focus specifically 
on TOEFL but also studies that focus primarily on other Instruments while 
providing data that relate performance on those instruments to TOEFL. 
Also, the 1973 and 1981 versions of the TOEFL manuals are summarized here 
because they present statistical data gathered over teveral large-scale 
administrations of the test* The 1973 version presimts data for the 
five-part test (which was in use before 1976; see below), and the 1981 

version presents data for the current three-part test. (A 1983 version of 
the manual is now available but is not summarized here, as it was not 
available at the time these summaries were compiled.) Also included are a 



few publfBhed papers that present analyses of TOEFL or describe its 
history, in order to provide some perspective on the basis for, and 
procedures followed in, development of the test. This collection has been 
prepared for use by researchers, test developers, foreign student coun- 
selors, admissions officers, teachers of English as a second language, and 
others who use TOEFL and thus might have an interest in summaries of 
reports relating to the test. 

The collection is a result of a larger effort to Identify all papers 
related to TOEFL that have been written in English, which has resulted in 
a comprehensive bibliography. This bibliography is to be published in the 
Modem Language Joumsl , Volume 67, No. 4, Spring 198A, under the title "A 
comprehensive TOEFL bibliography, 1963-1982." In developing this bibliog- 
raphy, the authors have identified relevant papers through the following 
means: 

1. A computerized literature search was conducted in December 1981 
and updated in January 1983. Data bases for this search included the 
Educational Resources Information Center (ERIC) system. Psychological 
Abstracts, Dissertation Abstracts International , Language and Language 
Behavior Abstracts . Social Science Citation Index , the Modern Language 
Association International Bibliography on Languages and Linguistics , and 
the ACTFL Annual Bibliography of Books and Articles on Pedagogy in Foreign 
Languages . Vol. 1 (1968)— Vol. 8 (1975-76). 

2. In April 1982 a letter Inquiring about local studies or other 
research involving TOEFL was sent to all directoirs of ESL teacher-training 
programs listed in C. Blatchford (Ed.), Directory of Teacher Preparation 
Programs in TESOL and Bilingual Education 1978-1981 ^Washington, P.O.; 
TESOL, 1979). 

3. In May 1982 a letter of inquiry was sent to 350 specialists In 
second language testing. This list included all recipients of Language 
Testing Notes , the newsletter of the Testing Commission of the Inter- 
national Association of Applied Linguistics (AIIA). 

4. Development of the bibliography was announced at the Language 
Testing Research Colloquium held at the sixteenth annual TESOL convention 
in March 1982, with a request for assistance in identifying all relevant 
studies. 

5. During 1982 a call for information regarding relevant studies 
appeared in the notes and news sections of several professional journals. 

6. A draft bibliography was sent in March 1983 for inspection to 
some 70 specialists in language testing listed on the Language Testing 
Research Colloquium mailing list with a request for information regarding 
any missing entries. 

In addition, all papers Identified in the search were examined for 
reference to other papers that might be appropriate. 
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Anong the papers that have been identified through this process, 
research papers falling into the following categories have been Included 
in the present collection: (a) published papers (i.e., journal articles, 
papers in edited volumes or published proceedings, and so forth): Xb) 
doctoral or master's theses; (c) papers included In institutional techni** 
cal report series; and (d) papers presented at professional meetings- 
Although many other unpublished reports have also been identified through 
the search and are included in the published bibliography, the research 
papers in the present collection are limited to those falling into one of 
the four categories listed above. There are undoubtedly other unpublished 
studies that also deserve consideration, but space and resource limita*- 
tions dictated that this collection be restricted. It was decided that 
the selection of studies to be Included in this collection should be based 
on objective criteria as opposed to a subjective assessment of quality by 
the three researchers who compiled this report. It was reasoned that the 
papers included here are more likely to have been subjected to some form 
of review, by peers and other professionals, and this review was viewed as 
exercising a form of quality control similar to that which might be 
exercised in a subjective selection process. Peer review and evaluative 
feedback is common in the preparation of articles for publication in 
professional Journals, and other media, and is also obtained in the 
process of delivering a paper at a professional meeting. Similarly, the 
writing of a thesis or dissertation also involves receiving and acting 
upon evaluative feedback. On the whole, the application of the above- 
mentioned criteria is believed to have provided a reasonably effective way 
of selecting appropriate research papers. ' 

Because the primary focus of this collection is upon resejarch, only a 
few nonresearch papers are summarized. Selected from a larger set of 
published descriptive or analytic papers about TOEFL, these are papers 
that appeared to provide the most extensive discussion about the history, 
development, or use of TOEFL. 

In a few instances, two different papers dealing with data from a 
given study were identified — for example, a thesis as well as a book 
chapter based on the thesis. In such cases, only one of the two papers is 
summarized in thip collection. 



TOEFL 



From 1963 to 1976, TOEFL consisted of five subtests: (a) Listening 
Comprehension, (b) English Structure, (d) Vocabulary, (d) Reading Compre- 
hension, 4nd (e) Writing Ability. Since the Seig;ember 1976 International 
administration and since the 1977 Institutional adQinistrations (see 
definitions below), TOEFL has consisted of three sections: (a) Listening 
Comprehension, (b) Structure and Written Expression, and (c) Reading 
Comprehension and Vocabulary. The two forms of the test are described in 
detail below. 



The Five-Part TOEFL 



The five-part TOEFL consisted of 200 total items (questions) and 
required 2 hours at^ 20 minutes of administration time. The five parts 
of the test were as follows; 

Listening Comprehension (50 items, 40 minutes) mfijisored the exam- 
inee 'TTi^TTTtyTounHerstana spoken English. The oral information was 
presented via tape recorder. This section consisted of three parts. In 
the first part (20 items) the examinee heard short questions or state- 
ments, then responded by indicating which of four printed alternatives 
best answered the questioner paraphrased the statement heard. In the 
second part (15 items), the examinee heard a short dialogue followed by a 
question about the dialogue and marked the printed statement that best 
answered the question. In the third part (15 items), the examinee heard a 
simulated university lecture eight minutes in length and was allowed to 
take notes. Terms that were used in the lecture were printed on a sepa- 
rate page that was provided for note taking. The lecture was followed by 
several questions and, after hearing each question, the examinee selected 
the best answer to the question from four printed alternatives. 

English Structure (40 items, 20 minutes) measured the examinee's 
mastery of important structural and grammatical points in spoken English. 
In each item, a short printed conversation between two speakers, part of 
which had been omitted, was presented. The examinee chose, from four 
printed alternatives, the word or phrase that correctly completed the 
conversation. 

Vocabulary (40 items, 15 minutes) n»asured knowledge of word mean- 
ings. This section contained two types of items. For each it&a of the 
first type (15 items), a sencence vas presented in which one word had been 
omitted, and the examinee selected, from the four alternatives provided, 
the one word that best completed the sentence. In each item of the second 
type (25 items), a phrase was presented and the examinee selected, from 
four options, the word or phrase with most nearly the same meaning as the 
given word or phrase. 

Reading Comprehension (30 items, 40 minutes) measured the examinee's 
ability to read and understand English prose, including the ability to 
make inferences and draw conclusions. Several passages were presented, 
and the examinee answered several four-option multiple-choice questions 
based on each passage. 

Writing Ability (40 items, 25 minutes) tested the examinee's ability 
to recognize effective style and appropriate usage and diction in written 



*In certain test administrations, additional items were included as 
experimental items for purposes of test equating. These added items were 
not part of th» basic test, however, and were not counted in deriving the, 
examinee's score. 
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English. This section contained two types of Items. For each Item of 
the first type (25 Items)* the examinee read a sentence in which four 
different words or phrases had been underlined, each marked with a differ- 
ent letter from A to D. The examinee selected the letter corresponding 
with the word or phrase that would not be accepted in standard written 
English. In each item of the second type (15 items), the examinee read a 
sentence containing a blank ^rtiere a word or phrase had been deleted. The 
examinee chose, from four alternative words or phrases, the one that best 
completed the sentence. 



The Three-Part TOEFL 

This form of TOEFL, which is currently in use, consists of 150 items 
and requires 1 hour and 45 minutes of actual testing time. (As with the 
five-part TOEFL, these figures apply to the basic test; additional items 
are included in certain administrations for purposes of test equating.) 

The ,three-pa»t TOEFL is similar to the five-part test, except that 
certain sections have been combined, and certain types of items within 
sections have been deleted or revised. (See Summary No. 62, Pike, 1979, 
for discussion of the empirical basis for revision of the test.) 

Listening Comprehension (50 Items, 35 minutes) measures the ability 
to underistand English as it is spoken In the United States. This section 
contains! three parts, with the oral material presented via tape recorder 
in each J>art. For each item in the first part (20 items total), the 
examinee hears a short statement ar.d then ctwoses the printed sentence 
(of four) that is closest in meaning to the one heard. For each itcsa in 
the second part (15 items total), the examinee hears a stwrt conversation 
between two speakers, which is followed by a <|be8tion asked by a third 
voice. The examinee must choose, from four printed phrases, the one that 
best answers the question heard. For each item In the third part (15 
Items total), the examinee hears a short talk, conversation or presenta- 
tion, such as a simulated news broadcast, short lecture, or public an- 
nouncement. The speech segment is follcwed by several orally presented 
questions and, for each question, the examinee must choose, from four 
printed phrases, the best ansv«r to the question. 

Structure and Written Expression (40 items, 25 minutes) measures 
mastery of Important structural and grammatical points in standard written 
English. The language used in this section is more formal than that of 
the Listening Comprehension section, and the topics are of a general 
academic nature. This section consists of two parts. For each item In 
the first part (15 Itons total), the eX4ipinee reads a sentence from which 
a word or phrase has been deleted. The examinee must ctoose, from four 
wrds or phrases, the one that best completes the sentence. For each itcon 
In the second part (25 items total), the examinee reads a sentence In 
which four words or phrases are underlined, marked A, B, C, and D. The 
examinee must identify the one underlined word or phrase that would not be 
accepted in standard written English. 
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Reading Conprehenaion and Vocabulary (60 items, 45 minutes) oeaaures 
the ability to understand tne meanings and uses of words as well as the 
ability to understand a variety of general reading materials. For each 
item in the vocabulary part (30 items total), the examinee reads a sen- 
tence in which one word or phrase is underlined. The examinee mist then 
select, from four words or phrases, the one that best preservss the 
meaning of the sentence if substituted for the underlined word or phrase. 
In the reading comprehension part (30 iteas total), the examinee reads a 
variety of short passages, each of which is followed by questions about 
the meaning of the passage. For each question, the examinee must choose 
the printed word or phrase that best answers the question. 



The Testing Programs 

TOEFL is administered under four separate testing programs: the 
International Testing Program, the Special Center Testing Progrcmi, the 
Institutional Testing Program, and the Overseas Institutional Testing 
Program. Under the International Testing Program, TOE^L is given at test 
centers around the world on six regularly scheduled Saturday test dates a 
year. Most applicants who need TOEFL scores for application to study in 
the United States or Canada take the test under this program. Interna- 
tional test ^^ministrations are provided both at overseas testing centers 
(i.e. , those outside the United States and Canada) and at d<»sestic testing 
centers (those within the United States and Canada). 

TOEFL is offered under the Special Center Testing Program six times 
a year, on regularly scheduled Friday test dates, in those months in which 
International Testing Program administrations are not off«:red. Like 
International administrations. Special Center administrations are offered 
at both overseas testing centers and domestic testing centers. 

In the Institutional Testing Program, test forms previously used in 
the International and Special Cc^nter testing programs are made available 
to institutions. The Institutions administer the test to their studenf*!, 
primarily for the purpose of placing the students at appropriate levels f 
an English language program or for determining the students' need for 
additional work in English. This service is provided in the United States 
and Canada. In other countries a similar service is provided through the 
Overseas Institutional Testing Program. 



The preceding sections provide background relating to the administra- 
tion of TOEFL that is necessary for understanding the nuethodology of the 
studies summarized in this collection. He now turn to a description of 
the summaries, with a brief overview of their structure and content. 



Nature of the Summaries 




In each of the siunaarles, we have sought to present a reasonably 
accurate recapitulation of the points made in the original paper. 
The reader is apprised that these summaries are not Intended to be evalua- 
tive. Each summary has been written from the perspective of the author or 
authors, with the details of method, results, and conclusions sumaarlzed 
as presented by the author. Thus, all statements of Interpretation or 
conclusion presented here are those of the author . It is left for the 
reader to draw his or her own conclusions as to the quality of each 
individual study. Because we are employed by the organization that 
publishes TOEFL, we felt that it might be inappropriate to evaluate 
critically the studies contained herein. However, we have occasionally 
Interjected comments; where we have done so, we have enclosed them In 
square brackets to differentiate them from the simunary of the author's 
statements. These observations are generally not of an evaluative nature, 
however, but are merely designed to note aspects of each study— usually 
methodological details— that may be unclear or missing from the report. 

It would be useful for the reader to keep in mind, in examining these 
summaries, that some studies may be more thorough than others* Thus, 
whether a given study's results lend themselves to firm conclusions may 
depend on a number of aspects of the methodology employed in the study. 
For each study summarized here we have attempted to provide detail that 
might aid the reader in this regard, including the size of the sample, the 
nature of the sample, the clrcu asnances of test administration, and so 
forth. But it must be reen[.haslzed that this collection is not intended 
to serve the function of a critical review. Rather, its obi«<etive is to 
call to the reader's attention the studies that have been done involving 
TOEFL (as well as some background papers), and to give a general overview 
of each study's method and restilts. The reader who wishes to look further 
into these studies is encouraged to read the original papers; the classi- 
fication scheme offered here should be of help in identifying studies of 
particular interest. 

1 

Many of the studies summarized here focus specifically on TOEFL. In 
some of the studies, however, TOEFL has played only a minor role. In the 
latter cases, we have focused principally on aspects involving TOEFL and 
have provided a relatively brief discussion of other aspects. Some 
studies, particularly theses, have presented extensive introductory 
material and reviews of the literature; in such cases, we have generally 
focused on the empirical aspect of the papers, condensing the Introductory 
material into short statements of background. 

In almost every swnmary^ we Indicate which version of TOEFL was 
used, the five-part or the three-part. In some of the reports summarized, 
however, the auttor did not indicate which version was used and, in these 
instances, we have indlcafed In brackets our assumption as to which was 
used, usually based on the o*#te of testing. Also, where relevant informa- 
tion was available, we report the type of administration that served as 
the source of TOEFL scores reported — i.e., (a) International, (b) Special 
Center, (c) Institutional, (d) Overseas Institutional, or (e) experi- 
mental administration. Where the source was not presented and the report 
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simply indicated that TOEFL scores were available (for students now 
enrolled In a U.S. college or university), we indicate in brackets our 
assumption that the TOEFL scores were derived from International or (after 
1977) Special Center administrations prior to matriculation, since such 
administrations are the principal sources of TOEFL scores in institutional 
records . 

Where iipformation was provided about tests or other instruments 
used, this information has been summarized. In some reports a test was 
not described, but a description of the test often could be found in 
another report. In such cases, the reader is referred to the summary of 
the latter report for a test description. 

In a few Instances the format of a test nay have changed over time. 
Wherever a test description is provided in a summary here, it is intended 
to summarize the author's description of the test at the time the descrip- 
tion was written. Also, the names of some tests mentioned in reports 
summarized here may have changed since the reports were written. Wherever 
this is true, the name in use at the time the report was written is used 
here. A notable example is the Graduate Record Examinations Aptitude 
Test, which, in 1982, was renamed the Graduate Record Examinations General 
Test. The original name Is retained throughout this collection, as it was 
the name in use when the studies employing the test were conducted. 

One measure that has been used frequently throughout these summaries 
is grade-point average (GPA). This measure has not been explicitly 
defined in many reports, perhaps because common conventions are used in 
computing GPA and authors have not thought it necessary to provide ex- 
planation. Typically, students in U.S. colleges and universities are 
given letter grades A, B. C, D and F, which are assigned numbers 4, 3, 2, 
1, and 0, respectively. A student's GPA for a given period of time (e.g., 
semester, year) is a weighted average of all grades assigned in that time 
period, where the weight for a given grade is the numbet of credit hours 
associated with that grade. Although there are variations to this system 
(e.g., pluses and minuses attached to the letter grades are taken Into 
account; certain courses are not counted), the basic concept of the GPA is 
assumed to be the saae In all studies in which GPA is used as a variable. 

Each sujmaary begins with the bibliographic reference to the report on 
which the summary was based. The style of the bibliographic reference 
follows that recommended in American Psychological Association (1983), 
Publication Manual of the American Psychological Association (3rd ed.). 
Washington, DC: Author. The headings used in the svmmarles were devised 
for use In this collection and follow the style typically used in research 
papers. Headings used for empirical papers here are Purpose , Method , 
Results , and Conclusions (or Results and Conclusions as a single section). 
An additional section, entitled Background , Is Included in cases, such as 
theses. In which the report includes extensive introdt»ctory material. 
Other headings are also used in a few other Instances. In the case of 
descriptive papers and the two versions of the TOEFL manual, a short 
Purpose section is followed by a single section entitled Discussion . 
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In some stfomaries, one or toore papers are cited In the text. Biblio- 
graphic references for the cited papers are provided in footnotes. If no 
bibliographic reference is provided, the paper Is among those summarized 
In this collection, and the reader can turn to the appropriate summary for 
the bibliographic reference. 

We believe this collection to be a nearly complete set of summaries 
of reports written prior to 1983 that fit the above-mentioned criteria. 
The TOEFL program is Interested in studies relating to TOEFL that may be 
conducted in the future and would like to request the author of any such 
study to send a copy of his or her article to the Research Coordinator, 
TOEFL Program Office, Educational Testing Service, Princeton, NJ, 08541- 
0001, USA. 



Classification of Reports 



General Categories 

The reports summarized here fall into logical groups, such as (a) 
studies of TOEFL 's relation to similar, standardized tests of Engliuh 
proficiency; (b) Its relation to other Indices of language proficiency; 
(c) its relation to later acadoaic performance, and so forth. Presented 
below is a classiflcatlcn scheme that lists the principal categories into 
which the summaries can I placed. Below each category name are several 
numbers, which refer to reports summarized in this collection. The 
numbers under a given category name indicate the reports In which the 
principal issue under study relates to that category. In the case of a 
fcsw reports, two or sometimes three Issues caaprlse the principal focus of 
the report, and In theye cases the report is listed under both or all 
three categories. Each general category is described belw by a phrase 
with a few examples. For more comprehensive definitions of the categories 
the reader is referred to the subsequent section, "Index to Reported 
Results." 

A The relation of TOEFL to similar, standardized objective tests of 
English language proficiency (e.g., ALI/GU, CELT, KTELP— see list of 
abbreviations below) 

Reports: 24, 36, 42, 51, 58, 59, 68, 81 

B The relation of TOEFL to other measures of English language profic- 
iency (e.g., cloze test. Interview, essay) 

Reports: 19, 20, 23, 25, 29, 30, 32, 38, 39, 44. 47, 48, 49, 52, 53, 
54, 62, 63. 67, 74 

C The relation of TOEFI to tests of Intelligence, acadcaalc aptitude, 
and achievement (e.g.. coBnK>n admissions tests such as ACT and SAT; 
reading achievement tests) 



Reports: 2. 6, 8. 24. 29. 61. 64. 65, 68. 69 
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0 The relation of TOI^L to later academic performance (''•g*, correlatioi 
cf TOEFL with college grade-jMilnt average, or GPA; combination of 
TOEFL with other variables in predicting CPA) 

Reports; 1, 4, 5, 11, 1^, 15, 17, 33, 35, 36, 37, 50, 55, 59, 66, 68, 
70, 71, 72, 75, 77, 78 

E Simple relation of TOEFL to student characteristics (e.g., native 
language, ccuntry, or region; se;^; educational level) 

Reports: 6, 9, 12, 1^ 18, 41, 43, 46, 79, 80 

F Complex relations involving student characteristics — i.e., differences 
In patterns, of results involving TOEFL as a function of student 
chars'* ^rlstics (e.g., native language, country, or region; sex; 
educaf evel) 

Reports: *J, 26, 51, 76, 80 

G Statistical analysis involving TOEFL (-^.g,, corxelatlons among t'.a 
TOEFL subtests; factor analysis involving TOEFL) 

Reports: 3. 21, 22, 26, 27, 3. 33, **2, 56, 76 

H Kiscellap;.-ou8 research isp.ui s (e.g., eff£*ts of llv5n>^ environment on 
TOEFL performance; relation of TOEFL to admissions i;c.;lslon8) 

Reports: 14, 31, 34, 73 

1 General descriptive paperc (e.g., history and devflopment of TOEFL; 
TC£FL manuals) 

Reports: 7, 27, 28, 40, 45, 57, 60, 82 



Index to Reported Results 

The above classificati n scheme lists each paper according to the 
category that comprises the principal focus of that paper, as it relates 
to TOEFL. In many of the reports summarized, however, TOEFL-related 
results that are secondary or incidental to the main focus are also 
presented. The following listing identifies all instances in which 
results pertaining to TOEFL are presented, whether or not they bear on the 
principal issue under study* For this purpose, « detailed subclassifica- 
tion scheae has been developed. Under each general C£::egory Heading, a 
list of subcategories is presented, each of which is designated by a 
letter'-'number cocRbination. Following each subcategory name (or following 
each example where there are two or more examples within a subcategory) 

are listed the numbers assigned to reports whose sunmaries contain data 
relevant to that subcategory. This listing thiu» serves as an index to 
reported results involving TOEFL. As a further aid to the reader, each 
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iadlvldual summary contains <ln the upper-right corner of the summary's 
first page) the letter-number combinations pertaining to all relevant 
subcategories of results presented in that sunmary. 

A The relation of TOEFL to similar, standardized objective tests of 
English language proficiency 

Al commonly used tests: ALI/GU (4); CELT (42); MTELP (1, 24, 59, 
68) 

A2 miscelJaneous tests: experimental objective tes^s (62); Lado 
tests (17); SC/GCE (58); TOEXC (81); UCB/EFL (51); WAIS-R Vocabu- 
lary test (36) 

B The relation of TOEFL co other measures of English language pro- 
ficiency 

Bl cloze test (23, 25, 29, 39, 44, 54, 62, 65, 66, 67) 
B2 interview* (1, 19, 20. 32. 38, 52, 62) and TSE (19, 20) 
B3 essay tests (58, 62, 63) 

B4 grades in fiSL classes (37, 42, 43); teachers' or other faculty 
members' ratings of English proficiency (24, 47, 68, 77) 

dictation test (44); noise test (32, 74) 
B6 rewriting task (30, 62) 

B7 miscellaneous measures: article usage indices (49); editing test 
(53); experimental oral listening comprehension test (48); Holtz- 
man Stress-unstress test (38); IMOF (38); oral doze test (74); 
pl&cement in ESL courses (42) 

C The relation of TOEFL to tests of intelligence, academic aptitude, 
and achievement 

CI common admissions tests: ACT (9, 24, 46); ®!AT (64, 80); GRE (6, 
8, 11, 70, 71, 80); LSAT (68); SAT (2, 8) 

C2 reading tests: Informal Reading Inventory (65); Iowa test (61); 
McGraw-Hill test (29, 61); Nelson-Denny test (61); Ferklns-Yorio 
test (29) 

03 miscellaneous tests: ITED (69); Raven's Progressive Matrics Test 
(29); SC/GCE achievement tests (58); TSWE (8) 
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D The relation of TOEFL to later academic performance 

Dl alnple correlation of TOEFL with grade-point average (CPA) 
(1, 4, 5, II, 13, 15, 17, 23, 33, 35, 36, 37, 40, 43, 50, 51. 55, 
66, 68, 70, 71, 72, 75, 78) 

. ♦ 

D2 combination of TOEFL with other variables In predicting CPA 
(5. 11, 15, 33, 68, 70, 72, 75, 78) 

D3 olscellaneous:- relation of TOEFL to Instructors* ratings of 
performance (77); telatlon of TOEFL to self-reported CPA (64) 

E Simple relation of TOEFL to student characteristics 

El native language, country, or region (3, 26, 41, 47^ 62, 75, 76) 

E2 sex (12, 41, 43, 47, 55, 79) 

E3 educational level (1, 8, 12, 69. 79) 

E4 major area of study (43) 

E5 native American vs. foreign (9, 46); performance of native Ameri- 
can stiulents (6, 18) 

E6 miscellaneous characteristics: age (1); currently U.S. citizen 
vs. non-U.S. citizen (80); deaf vs. hearing parents, of deaf 
examinees (16); English as primary vs. second language (80); 
ethnic group (12, 55); occupation in home country (77); parents* 
education (55); planni^ degree (47); previous gr^es (1); reported 

vs. did not report TOEFL scores to institutions (79); social 
adjustment (69); TOEFL repeater vs. nonrepeater (79); TOEFL taken 
in foreign vs. domestic center (79); type of secondary school 
attend^ (55) 

F Complex relations involving student characteristics— i.e. , differences 
in patterns of results involving TOEFL as a function of student 
characteristics 

Fl native language, country, or region (3, 4. 10, 26, 51, 59, 62, 
64, 76) 

F2 sex (43, 47, 51, 78) 

F3 educational level (4, 35, 51, 75, 78) 

F4 major area of study (35, 43, 51, 70, 78) 

F5 isltcellaneous characteristics: English as primary vs. second 
language (80); planned degree (47); sponsored vs. nonsponsored 
(17) 

» 

18 

ERIC 



-13- 



6 Statistical analysis involving TOEFL 

Gl correlations among TOEFL subtests (1, 20, 27, 28, 33, 42, 48, 58, 
62, 63. 74) 

G2 factor analysis involving TOEFL (26, 33, 42. 56, 58, 69, 74, 
76) 

G3 test equating (21. 22) 
G4 item coaparisons (3, 9) 



H Miscellaneous research issues: characteristics of TOEFL candidates 
(79); effects of instrwrtion on TOEFL (33. 41, 52, 68, 77); effects of 
iten disclosure on TOEFL (34); effects of living environment on TOEFL 
perfonaance (46, 73); relation of actual TOEFL scores to self-reported 
TOEFL scores (64); relation of TOEFL to admissions decisions (14, 31); 
relation of TOEFL to self-evaluation of English proficiency (4) 



I General descriptive papers 

11 history and development of TOEFL (7, 45, 57, 60) 

12 TOEFL manuals (27, 28) 

13 the role of TOEFL in admissions decisions (40. 82) 
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Abbreviations Used 



Throughout this collection, certain abbreviations are used. Some of 
these abbreviations, such as letters representing statistical terms, are 
80 common that they are presented without definition wherever they are 
used. Most other abbreviations used here are acronyms of teste. These 
abbreviations are defined at the beginning of any summary In which 
they appear. However, It is useful also to present a list defining all 
abbreviations and acronyms, so that the reader may refer to this list as 
a reminder when necessary. The following list presents the full names 
associated with all abbreviations and acronyms used in ti|« summaries. 



Statistical symbols and terms 



E. *** £ ^^^^t generally from analysis of variance 

£ — general factor 

KR-20 — Kuder-Rlchardson reliability, formula 20 

KR-21 — Kuder-Richardson reliability, fonaula 21 

II — ' number of subjects (If presented in parentheses after a 
statistic, this indicates the nuaber of subjects on which 
the statistic was based.) 

ns not significant 

£ — probability level, or significance level 

jr — Pearson product-moment correlation 

K — multiple correlation 

SO — standard deviation 

£ — Student's t^, generally used as test of difference between 
means 



General Terms and Organizational Acronyms 

AACRAO — Ameifican Association of Collegiate Registrars and Admissions 
Officers 

AID — Agency for International Development of the U.S. Department 
of State 
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CAL — Center for Applied Unguis ties 

CEEB College Entrance Examination Board inmi the College Board) 

BFL Engl^ u as a foreign language 

ESL — ^iish as a second language 

ETS — Educational Testing Service 

CiPA — grade-point- average 

GREB — Graduate Record Examinations Board 

IRT — item response theory 

JAN — Judgment Analysis Technique 

Acronyms for Tests and Other Measures 

ACT — American College Test » 

ALI/GU — Test of the African Language Institute at Georgetown University 

CELT Comprehensive English Language Test for Speakers of English as 

a Second Language 

CESL Test of the Center for English as a Second 
(SIU) Language at Southern Illinois University 

ELSA — English Language Skills Assesasent in a Reading Context 

EPE — English Placement Examination 

ESLAT — English as a Second Language Achievement Test 

FSI — Foreign Service Institute; used here in reference to the FSI 
interview, which is currently referred to as the Interagency 
Language Roundtable (ILR) interview 

GMAT — Graduate Management Adtoission Test 

ORE — Graduate Record Examinations 

IMOP — Indirect Measure of Oral Output 

ITED — Iowa Test of Educational Developnent 

LCPT — Listening Comprehension Picture Test 

LSAT Lav School Admission Test 
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MTELP — Michigan Test of English Language Proficiency 

PAA — Prueha de Aptitud Acadealca 

SAT — Scholastic Aptitude Test of the College Board 

SC/GCE — Examination for the School Certificate and General Certificate 
of Education of the West African Examinations Council 

SLEP — Secondary tevel English Proficiency Test 

STEL — Structure Tests for English Language 

TOEFL — Test of English as a Foreign Language 

TOEIC — Test of English for International Comnajnicatlon 

TSC — Tennessee Self-Concept Scale 

TSE — Test of Spoken English 

TSWE — Test of Standard Written English 

UCB/EFL — University of California, Berkeley, Test of English as a 
Foreign Language 

WAIS-R — Wechsler Adult Intelligence Scale, Form R 
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SUMMARIES Al, B2, Dl, E3, E6, Gl 

1. Abadzl, J. (1976). Evaluation of foreign students [sic] admission 
procedures used at the University of Alabama (Doctoral disserta- 
tion, University of Alabama, 1975). Dissertation Abstracts 
International , 36, 77S4A. (University Microfilms No. 76-13, 
884} 



Purpose 



As part of an examination of the admissions process for foreign 
students at the University of Alabama, this study investigated the rela- 
tion between academic performance at the university and several variables, 
including the score on TOEFL and on other measures of English proficiency. 
Also exottlned i#ere effects of such background variables as age, country of 
origin, undergraduate vs. graduate status, and major field* Of many 
analyses, those pertaining specifically to TOEFL are emphasized In this 
summary. 



Backgroimd 



Previous research has suggested a relationship between some of the 
above-mentioned variables and academic success. The few studies located 
relating academic performance to tests of English proficiency showed 
generally low correlations. Studies relating foreign students' academic 
success to their performance on American admissions tests also showed 
relatively low- correlations. Research using TOEFL as a moderator variable 
along with admissions tests has often yielded inconclusive results. 

Method 



The subjects were 70 foreign students (60 males and 10 females) who 
entered the University of Alabama in fall 1974 (N - 37) or spring 1975 
(N^ 33). Fifty-three were tmdergraduates and 17 were graduate students. 
The subjects ranged in age from 17 to 36 years and represented 28 differ- 
ent countries. Several different major fields were represented, with the 
modal number of subjects majoring in engineering (£ » 34). The mean total 

TOEFL score of the sample was 548, with mean subtest scores ranging from 
52 to 57. 

Individual interviews were conducted during the week precedir^g 
registration with the 31 subjects who had Just arrived in the United 
States. (The others had been in the United States for several months or 
years.) These subjects were asked some general questions then listened to 
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a slBulated lecture of approximately 180 words and recalled what they 
understood of It. Approximately two months later a second interview was 
conducted with thi^se 3i sid>jects, and the same simulated lecture was used. 
The subjects w«re rated by two Judges on three five-point scales: fluency 
(F), pronunciation (P)» and comprehension (C) (i.e., recall of the 
passage). Agreement between judges was high, as Che coefficients of 
correspondence ranged from .93 to .99 for the six scores (i.e.* three 
scales for each of two administrations) • 

All but eight of the 70 subjects took the Michigan Test of English 
Language Proficiency (MTELP) on arrival at the university. The eight 
subjects who did not take the MtELP were assigned the mean scores for 
their countries on this test for purposes of data analysis. A composite 
score based on Che three subtests was used in the data analyses. 

The MTELP is a multiple-choice test consisting of three sections: 
Grammar, Vocabulary, and Reading Comprehension. In the Grammar section, 
each item consists of a statement with a deleted word or shoxt phrase, and 
the examinee must choose, from four alternatives, the missing word or 
phrase. In the Vocabulary section, each it&n is either a sentence with a 
deleted word or a sentence with an underlined word, and the examinee must 
identify, frtxn four choices, the deleted word or a synonym for Che under-* 
lined word. The Reading Comprehension section consists of several reading 
passages, and each passage is followed by several multiple**cholce ques- 
tions testing factual understanding or inference, (description para- 
phrased from a recent publication related to the MTELP] 

Data from the five-part TOEFL [presumably from International adminis- 
trations] were available for 25 of the subjects (TOEFL scores were gener- 
ally not available for students who had previously attended institutions 
In Che United States or Britain or for certain other students). 

The grade-point average (GPA) and number of credit hours taken in 
the first semester were tabulated for each of the 70 subjects. The GPA 
and credit hours in the second semester were tabulated for each of the 33 
subjects who entered in fall 1974. For all 70 subjects, an estimation of 
previous academic performance [presumably performance in the institution 
the student had just attended, whether in the United States or other 
country] was obtained by transforming the subject's grade into a number on 
a six-point scale equivalent to the U.S. grades A, B-f, B, C4-, C, and D. 
Also, for the 19 subjects who had transferred to the university from a 
junior collie, the average junior college grade was calculated. 

All of the above-mentioned variables were submitted to correlational 
analysis, along with two other variables: age and undergraduate vs. 
graduate status. 
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Results 



Correlations of the various measures used in the study with the TOEFL 
subscores, TOEFL total score* and MTELP score are presented in Table 1; 
only those correlations exce^ing .30 are reported. Note that the TOSFL 
subtests, abbreviated in the table, are Listening Comprehension (LC), 
\ English Structure (ES), Vocabulary (V), Reading Comprehension (RC), and 

Writing Ability (VA). 

In addltiolll analyses were conducted to determine the ef ects of 
geographic area, major field* and classification (undergraduate, graduate, 
transfer froo abroad, transfer frras junior college) (!N «■ 70 in each case). 
These analyses revealed differences as a function of The variables listed, 
particularly with regard to first-semester CPA. Graduate students had the 
highest mean CPA (2.46) and junior college transfers the lowest mean CPA 
(1.12). Among muijor field groups, science majors had the highest (iean CPA 
(2.21), perhaps because most science majors were gradtiate students. Among 
the countries representee!, subjects from Taiwan had a relatively high mean 
CPA (2.42), perhaps because they were almost all graduate students. 

Multiple regression analyses were performed involving several vari- 
ables; t)K>se involving TOEFL will be briefly mentioned. Prediction of the 
TOEFL score from the MTELP score and the interview ratings yielded a 
significant multiple R of .82. [The 21 for this analysis is unclear.] 
Prediction of the MTEXP score from the TOEFL subscores and total score 
yielded a significant multiple R of .85 (N - 23), with each subscore 
contributing substantially to the prediction. Prediction of first-* 
semester CPA from the TOEFL subscores and total score yielded a multiple R 
of .67, which was not significant, probably because of the low N (25). 

For the 31 subjects Interviewed, there was an Improvement in ratings 
over the approximately two-month period between interviews. For the 33 
subjects who had both first*- and second-semester CPA scores, a £ test 
showed the latter to be significantly higher than the former; also, the 
number of credit hours carried Increased significantly from first to 
secomi semester for these subjects. 



Conclusions 



Conclusions from this study must be regarded as tentative due to the 
•mall number of subjects involved. First-semester GPA correlated moder- 
ately highly with the total TOEFL score; also, the correlation between 
first-semester CPA and the MTELP score, although only .29 (and thus not 
appearing in Table 1), was significant. These data suggest that there 
is some relationship between English proficiency and initial academic 
performance. Correlations involving second-semester CPA were generally 
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Table t 

Correlations Involving TOEFL and the MTEI 
with Absolute Values of .30 or Greater 

4f ' 



Variable 


N 


LC 




V 


Age 


a 


-.73 






Grad. (0) vs. U'grad. (1) 


a 


-.54 






Semester 1 CPA 


a 




.45 


.32 


Semester 1 No. hours 


a 


.53 


.35 


.32 


Semester 2 GPA 


b 






.34 


Semester 2 No. hours 


b 






-.38 


Previous average grade 


25 




-.36 




Average grade— Jr. Coll. 


b 


.75 


.86 




Interview-F (admin. 1) 


b 


-.40 






Interview-P (admin. 1) 


b 


-.42 






Intervlew-C (admin. 1) 


b 


-.52 






Interview-F (admin. 2) 


b 


-.47 




-.30 


Interview-P (admin. 2) 


b 








Intervlew-C (admin. 2) 


b 


-.49 




-.30 


TOEFL LC 


25 




.35 


.33 


TOEFL ES 


25 








TOEFL V 


25 








TOEFL RC 


25 






.73 


TOEFL WA 


25 








TOEFL Total 


25 









TOEFL Subtest TOEFL 

lit 551 Total MTELP 



-.32 -.30 



.50 .52 .43 
.40 



-.38 

.66 .85 

-.59 
-.65 
-.64 

-.30 -.70 
-.60 
-.68 

.46 .62 .49 

.69 .67 .81 .64 
.69 .67 .81 .71 
.89 .72 
.80 .65 
.79 



*]J8 are 25 for correlations involving TOEFL and 70 for correlations 
involving the MTELP. 

^[Not reported are the numbers of tiubjects taking TOEFL among 
students for whom second-semester GPA data are available; among transfers 
from junior colleges; or among students who were intervlewd.j 
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lover than those involving first-semester CPA (even though the CPAs of the 
two semesters were relatively highly correlated with each other— £ - ,76). 
For correlations involving TOEFL this difference can be seen in Table 1; 
for the MTELP, the correlation dropped from a significant .29 to .00. 
These results suggest that, while foreign students' initial grades are 
related to their English proficiency, their grades become increasingly 
independent of their initial level of proficiency. 

The intercorrelations among TOEFL subsccres and TOEFL total ranged 
from .20 to .88 [note, however, chat the maximum £ shown in Table i is 
.89], suggesting that TOEFL is a well-structured te¥t, with the subtests 
showing some relationship to each other but, at the same time, measuring 
somewhat different aspects of English knowledge. The correlation of .79 
between the MTELP and TOEFL scores is consistent with earlier research 
showing a relatively strong relationship between these two measures of 
English proficiency. 
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9 AidertnAn D L. (1982). LaMuagc proficiency as a moderator variable 
2. Alderman. ^D.^.^tl^»^J. aptUudl. Journal of Educational Piycholo gy, 

74, 580-587, (Also TOEFL Research Rep. No, 10, 1981; ETS 8e- 
■^arch Rep. No. 81-411. Princeton, NJ: Educational Testing 
Service. 



Purpose 



This study examined the role of language proficiency in testing 
academic aptitude and, specifically, t? c td the hypothesis that proficien- 
cy In a second language acts as a motierator variable in accounting for 
verbal aptitude scores on tests given in that language. 



Method 



A total of 361 nonnative English-speaking students at three public 
and three private secondary schools in San Juan, Puerto Rico, participated 
m the study. In exchange for a fee waiver on the tests and a nominal 
stipend, in November 1980 each student took the Colleg«^ard Scholastic 
Aptitude Test (SAT), the Prueba de Aptitud Acadtfmica (PAA), and the 
three-part TOEFL. 

The SAT is a multiple-choice test written for applicants to U.S. 
colleges and universities. It consists of verbal and mathematical apti- 
tude sections and the Test of Standard Written English (TSWK). The SAT 
verbal section Includes analogies, antonyms, reading comprehension, and 
sentence completion items; the mathematical section measures ability to 
solve probleas involving arithmetic reasoning, algebra, and geometry. The 
TSWE is a test for native speakers of English that assesses knowledge of 
the conventions of standard written English; it contains questions involv- 
ing the correction of sentences and questions involving English usage in 
an error recognition format. The PAA contains verbal and mathematical 
aptitude sections parallel to those found in the SAT and a test entitled 
the English as a Second Language Achievement Test (ESLAT). The ESLAT is 
written for nonnative speakers of English and contains questions on 
grammar and reading comprehension. 

A regression analysis of SAT scores on PAA and TOEFL scores was 
performed to determine the significance of the interaction between the PAA 
and TOEFL in accounting for scores on the SAT. A significant Increase In 
explained variance with the addition of a product term representing this 
interaction would confirm the expectation that student proficiency in a 
second language ntoderates performance on the SAT. 
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ReauXts and Conclusions 



Table 1 displays the regressions of aptitude test scores obtained in 
a second language (SAT) on aptitude test scores obtained in a first 
language (PAA) and measures of second language proficiency (TOSFL, ESLAT, 
TSWE). The two left-most columns of values shov an increase in the 
multiple correlation (R) and explained variance i9r) as each independent 
variable is entered inTo the regression. The regression coefficients 
given in the table are the raw (b) and standardized (B) weights for the 
three independent variables. The final column indicates whether the 
inclusion of the dependent variable produced a significant increase in the 
amount of explained variance for the dependent variable. 

It is apparent that both TOEFL and the ESUT accounted for signifi- 
cant increases in the variance in SAT scores beyond that explained by 
parallel aptitude tests in the examinee's native language. TOEFL alone 
explained an additional 30 percent of the total variance on tne SAT-V and 
about 10 percent of the total variance on the SAT-M. This finding sup- 
ports the hypothesis that proficiency in a second language plays a signi- 
ficant role in scores on aptitude tests administered in the second lan- 
guage. Thus, when taking aptitude tests in English, students apparently 
need a fairly high level of proficiency in English to adequately demon- 
strate their aptitude. Scores on language proficiency tests such as TOEFL 
should be taken into consideration as institutions make foreign-student 
admission and placement decisions based on academic aptitude tests given 
in English. 
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Table 1 

Regressions of Aptitude Test Scores in Second Language (SAT) on 
Aptitude Test Scores in Fitst Language (PAA) and Measures 
of Second Language Proficiency 



Multipla 

Otpaadnt Corrclatlcmf Indcpindat MfrvsaioR Co«ffici«it* StatiaticaX Tm€ of 
Vwtobl* E VafUbl— e !5g Hodanitor gff«ct 



SAT-V 


.6655 
.8566 
.8934 


.4429 
.7341 
.7982 


PAA-V 

Toen.« PAA-V 


-.8303 
-.8329 
.0025 
490.8048 


-.97 
-.94 
2.57 


.10 
.14 
.00 


P (1,357) 


m 


113.39* 


tAT-V 


.6655 
.8022 
.8615 


.4429 
.6436 
.7423 


fAA-V 
ESLAT 

ESLAT« PAA-V 
Ccoa«t«ttt) 


-.7913 
-.7916 
.0021 
506.1330 


-.92 
-1.21 
2.75 


.09 
.10 
.00 


P (1,357) 


m 


136.70* 


SAT-V 


.6655 
.8790 
.8805 


.4429 
.7727 
.7754 


PAA-V 

TSWE 

TSWE* PAA-V 
(conatsnt) 


.0952 
2.2814 
.0071 
68.4927 


.11 
.23 
.58 


.08 
2.26 
.00 


P (:,357) 


m 


4.216 


SAT-n 


.7955 
.8514 
.8871 


.6328 
.7248 
.7870 


PAA-M 

TOEIX 

TOEFL*' PAA-M 
(eonttcnt) 


-.6207 
-1.2147 
.0026 
558.7010 


-.70 
-1.17 
2.56 


.11 
.16 
.00 


P (1,357) 


m 


104.26* 


SAT-M 


.7955 
.8259 
.8777 


.6328 
.6821 
.7703 


PAA-M 
ESLAT 

ESLAT K PAA-M 
(coDvtant) 


-.5502 
-1.0554 
.0022 
540.7829 


-.62 
-1.39 
2.66 


.10 
.11 
.00 


P (1,357) 


m 


137.16* 


$AT-M 


.7955 
.8615 
.8718 


.6328 
.7422 
.7601 


PAA-H 
TSWE 

TSWE" PAA-M 
(convtant) 


.0664 
-9.0i86 
.02X 
236.5505 


.07 
-.77 
1.51 


.09 
2.69 
.00 


P (1.357) 


m 


26.69* 



% < .01 
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3, Aldeman, D. L. . & Holland, P. W. (1981). Item performance across 
native language groups on tl.e Test otEW g llsh as a Foreign 
Language (TOEFL Research Rep. No. 9; ETS Research Rep. No. 
81-16. ) Princeton, NJ: Educational Testing Service. (ERIC 
iv^cument Reproduction Service No. ED 218 922) 



Purpose 



A special chi-square statistic was used to determine the sensitivity 
of TOEFL items to differences in native languages of ex^lnees. Differ- 
ences among six language groups were assessed for every item on esrh of 
two TOEFL foicts. 



Method 



The subjects were 12,379 examinees from six language groups: African, 
Arabic, Chinese, Germanic, Japanese, and Spanish. The African group 
combined four languages: Efik, Fanti, Ibo, and Yoruba; the Germanic group 
combined Danish, Dutch, German, and Swedish. The data for analysis were 
based on two International administrations of the three-part TOEFL, one in 
Novenber 1976 and the other in November 1979. 

In the principal analysis, data for each test were submitted to 
chi-square analysis to determine each Item's sensitivity to language 
differences. In addition, specialists in English as a second language 
(ESL) were asked to examine the results of the analysis based on the first 
of the two test administrations and (a) to suggest explanations for items 
with high sensitivity to language differences and (b) to identify items 
from the second test administration that they believed would be most 
sensitive to differences among the language groups. 



Results and Conclusions 



Approximately seven-eighths of the itens In each TOEFL form showed a 
significant d^ree of sensitivity to language differences according to the 
chl-aquare analysis. The items that were most sensitive were those in the 
Reading Comprehension and Vocabulary section, particularly in the first 
^st adnlnistration. It is suggested that knowledge of specific words in 
single sentences and reading passages may be more susceptible to linguis- 
tic contrasts than are either aural skills or syntactic rules. 

Subjects in each language group were placed into 10 strata according 
to their overall TOEFL scorep* Analyses showed that the relative advan- 
tages and disadvantages for different language groups it e relatively 
stable across strata. 
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Analyal* of overall test seores showed that the two language groups 
with the hlghe. test scores were the Germanic and Spanish groups, the two 
with the closest affinity to the English language. The rank order of 
language groups varied soa^what by test section, however; the two highest 
ranking groups for Listening Comprahenslon and for Reading Coaprehenslon 
and Vocabulary were the Germanic ant' Spanish groups, whereas the highest 
ranking groups for Structure and Written Expression were the Germanic and 
African groups. 

The ESL specialists, when asked to identify reasons for certain 
Items' sensitivity to language differences, found this task very diffi- 
cult, as they could only offer comments involving languages with which 
they were familiar. Further, when asked to predict which items in the 
second test form would be most sensitive to language differences, the 
reviewers were generally unsuccessful. Pr Miction was only at a chance 
level for Listening Comprehension and Structure and Written Expression; 
for Reading Comprehension and Vocabulary the success rate was somewhat 
higher but was still considered not reliable enough for practical applica- 
tion. An a priori contrastive analysis of this type appears to be quite 
speculatlveT 

The study demonstrates a statistical procedure that can help deter- 
mine the sensitivity of TOEFL items to language differences. It is 
cautioned, however, that results that are significant by this method do 
not provide a sufficient basis for excluding an item from the test; even a 
relatively small difference can yield a significant chl-square, and some 
degree of variation is to be expected, based on known contrasts with the 
English language. However, the method is useful in identifying items that 
yield exaggerated or unexpected deviations from expected item performance 
across language groups. 
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A. American Association of Collegiate Registrars and Admissions Officers. 

(1971). AACRAO-AID Participant Selection and Placement Study * 
Report to the Office of International Training, Agency for 
International Oeveloinent, U.S. Department of State. Washington, 
DC: Author. 



The U.S. Agency for International Development (AID) annually sponsors 
a program in which foreign nationals attend U.S. colleges and universities 
to prepare them to assist in the econc»alc and social development of their 
home countries. Several aspects of these students* selection and place- 
ment were examined. This suBimary focuses primarily on data dealing with 
prediction of academic performance, in which TOEFL plajred a role. 



The subjects were 1,004 AID progran participants, including the first 
regular participants arriving in 1967 and 1968 and 100 Vietnamese partici- 
pants attending a six-month Intensive English language program. Partici- 
pants were selected by AID, with home-government approval, partly on the 
basis of maturity and experience, and they were placed in U.S. colleges 
and universities by AID or another U.S. federal agency. 

Neat'^v three-fourths of the subjects were from the Far East or Africa 
(vs. one-Laird of the total foreign student population in the United 
States). Sixty percent of the subjects specialized in education, social 
science, or agriculture (vs. 21 percent of all foreign students). The 
subjects' median age wa& 28, 58 percent bad been out of school at least 
three years, and the vast majority held professional positions in their 
home countries. Thus, these subjects were different from the general 
population of foreign students in the United States. Nevertheless, the 
mean TOEFL score of the subjects (483) was nearly identical to the ijKan 
score of -11 foreign applicants to U.S. institutions between February 
1964 and April 1967 (484). The sample was 81 percent male (vs. 75 percent 
for all foreign students). 



Each subject cwnpleted a questionnaire indicating (a) year of birth, 
(b) number of years since last school attended, and (c> rank in class (top 
10 percent, top 25 percent, top 50 percent, bottom 50 percent). Each 
subject was also administered at least one of the following tests: TOEFL, 
tte English proficiency test of the American Language Institute at George- 
town University (AH/GU), and the College Board Scholastic Aptitude Test 
(SAT) for undergraduates, or the Graduate RecorU Examinations (GRE) 
Aptitude Test for graduate students. 



Purpose 



Method 
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The ALI/GU is a battery consisting of four tests: English Usage, 
Vocabulary and Reading, listening (all multiple-choice), and Oral Bating 
(interview). In the English Usage test, each item is a sentence with a 
word or phrase deleted, and the exaaioee mist clK>ose, from three alter- 
natives, the word or phrase that best completes the sentence. In the 
Vocabulary and Reading test, each vocabulary iten is either a sentence 
with a deleted word or a sentence with an underlined word, and the exam- 
inee oust choose, fron four alternatives, the correct deleted word or a 
synonym for the underlined word. In the reading part of this test, the 
examinee reads several short passages and, for each passage, answers 
several multiple-choice questions. For each item of the Listening teat, 
the examinee hears a stu>rt question or a short statement and must choose, 
from four printed alternatives, the correct answer to the question or a 
correct paraphrase of the statement. Idescription paraphrased from a 
recent publication concerning the ALI/GU test! Only these three tests 
were administered to the subjects in the present study on their arrival in 
the United States, and the ALI/GU score reported here is a composite of 
scores on these three tests. 

The SAT Is a multiple-choice test of verbal and mathematical aptitude 
for applicants to U.S. colleges and universltlee [see Summary No. 2, 
Alderman, 1982). The GRE Aptitude Test [recently renamed the GRE General 
TestJ is a multiple-choice test designed for applicants to U.S. graduate 
schools and measures verbal and quantitative aptitude. The verbal ability 
section tests the abilities to discern, comprehend, and analyse relation- 
ships among words or groups of words within sentences and written pass- 
ages. The quantitative section measures basic mathematical skills, 
understanding of elementary mathematical concepts, and ability to reason 
quantitatively and solve quantitative problems, [description paraphrased 
from a recent GRE publication] 

Three ratings of each subject's quality of academic performance In 
his or her home country were made, based on the subject's transcript: (a) 
quality in relation to home-country standards, as judged on a five-point 
scale by credentials analysts from the Aa^rican Association of Collegiate 
Registrars and Admissions Officers; (b) competitiveness of Institution for 
which the subject was suited, as judged on a four-point scale by the 
credentials analysts; and (c) quality in relation to standards at the 
subject's assigned institution, as judged on a five-point scale by receiv- 
ing admissions officers. The above-mentioned factors all served as 
predictor variables in the analyses t o be presented. ^ ^ 

The following indices, computed during the subjects' first year at a 
U.S. college or university, served as criterion variables in the analyses: 
(a) grade-point average (CPA), on a four-point scale, for first lemastar, 
second semester, and total first year; (b) credits earned during the first 
year; and (c) f Irst-semaster and first-year "achievement index," defined 
as CPA squared times number of credits earned. Cradumte students were 
also rated by their advisers on a five-point scale (a) relative to other 
foreign students in the discipline and (b) relative to all other students 
in the discipline. 
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Results and Conelusions 



Two Inieial results involving TOEFL are worth noting. First, the 
correlation between scores on TOEFL and the ALI/GU was .84, indicating a 
considerable aaount of similarity in these two tests. Second, when the 
subjects were asked if their English proficiency fas adequate for full- 
time study, 74 percent of those who said "yes" had TOEFL scores of 450 and 
above; 68 percent of those who said "no" had TOEFL scores belcw 450. 

Thus, the subjects' TOEFL scores tended to correspond with their percep- 
tions of their English proficiency. 



Prediction of Academic Performance-— Dndergraduates 

Correlations of predictor and criterion variables were significant 
but generally low. (Ns ranged from 260 to 413, as different data were 
available for different subjects.) Of the correlations between the 
predictors and the three CPA scores— first-semester, second-semester, and 
full-year CPA — those involving the full-year CPA were the highest in 
almost every case. In the results presented below, the term "CPA" refers 
to first-year CPA. The best predictors of GPA were the mathematicA 
section of the SAT (jr - .55), admissions officers* ratings of past aca^ 
demic record (.37), and birth year (.36), the last result reflecting 
better performance for younger subjects. 

Grade-point average was not predicted very well by either TOEFL 
(.25) or the ALI/GU (.23). The first-year achievement index, which 
combines GPA and credits received, correlated someu^at more highly with 
these tests (TOEFL: .36; ALI/GU: .32), perhaps because thoae subjects 
with low English test scores take more remedial courses and thus receive 
fewer regular credits. 

a 

It might be assumed that the low correlations were due to matching In 
quality of subjects and institutions. However, no clear relation was 
found between admissions officers* ratings of subjects* prior acadaalc 
performance and the selectivity of the subjects* assigned institutions. 

The correlations could also have been reduced by the pooling of 
subjects from many countries. Correlations were computed sci>arately for 
two major sending areas, Af rica and Vie tnam. For the Vie tnames e subje cts, 
co r relstlfthi 6f U P A w ltTi TOSFL and~^th the Att/crwere' both~."4y"(Ms licre 
between 127 and 227.); however, for the African subjects, the corr'elation 
of GPA with TOEFL was only .17 and with the ALI/GU was nonsignificant (and 
thus not reported), (Sa were between 116 and 121.) Although the basis 
for the difference between regions is not clear, apparently the value of 
these tests for predicting performance cannot be waid to be the same for 
all geographic areas. 
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Prediction of Aodealc Perforance for Graduate Studeata 

For the graduatflr«tud«iita» CPA atoifed a Gorrelatlon of .19 with TOEFL 
and .14 with the ALX/GU; correlation* with first-year achievement index 
were slightly higher (.33 and .30, respectively). Separate correlations 
were coaputed for fonr geographic sidsgroups» three curricular subgroups, 
two levels of English proficiency, and for subjects out of school for 
different lengths of time; none of these groupings resulted in Improved 

prediction of CPA. 

It is concluded that English proficiency scores did not have strong 
predictive value in this study. They appear to have been of greater value 
in indicating how heavy an academic workload a student should take. 
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Andalib, A. A. (1976) • The academic success of undergraduate Iranian 
students in selected Texas universities (Doctoral dissertation, 
East Texas State University, 1975). Oissertation Abstracts 
International , 36, 4881A. (University Microfilms No. 76-4618) 



Purpose 



This study assessed the degree to which the academic success of 
Iranian students in selected Texas universities was predicted by TOI^L as 
well as by five other variables: (a) Ao»rican College Test (ACT) scores, 
Mb) College Board Scholastic Aptltule Test (SAT) scores, (c) high school 
scholastic average, (d) age, and (e) years out of school. 



Background 



Examination of previous studies suggests that students* academic 
performance in high school is related to their college performance. One 
study (American Association of Collegiate Registrars and Adilssions 
Officers, 1971) suggests that TOEFL is of limited value as a predictor, as 
is the verbal SAT score, but that undergraduate grades are related to the 
mathematics score on the SAT and negatively related to age and years out 
of school. 



Method 



The Subjects were Iranian undergraduates at several Texas univer- 
sities. Of 336 students considered, 126 (117 males and 9 females) had 
data for three predictor variables: high school sclwlastic average, age, 
and years out of school; these students comprised the stud*' sample. In 
addition to data for the above three variables, 42 of the subjects had ACT 
scores (these subjects are called Group 1), 26 had SAT scores (Group 2), 
and 16 subjects had TOEFL scores (Group 3) Jail presumably obtained via 
preadmission testing; i.e.. International administrations, in the case of 
the TOEFL]. The total sample of 126 students is called Group 4. [Names 
and number of universities participating are not specified, nor are the 
class standings of the subjects.] The criterion variable to be predicted 
by the above-mentioned factors was the subjects* undergraduate grade-point 
average (GPA)^ defined according to the stardard five-point scale (i.e., A 
4, B - 3, C • 2, D • 1, F - 0). 

High school average was computed on a 20-point scale, based on the 
subjects* perforaance in grades 10-12. For the approximately 5 percent of 
the subjects who had attended high school in the United States, grades 
were transformed to the 20-point scale according to COTuaon guidelines. 
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Thc ACT !• « fflultlplft-cholce teit In four p«rtt. The Bnglith Uf«g« 
subtest aeasuret knowledge of punctuation, style, end other aspects of 
writing. Matheaatics Usage assesses aathcmatical reasoning ability. 
Social Studies Reading and Natural Science Reading measure reading and 
reasoning skills in these two subject areas. 

The SAT contains two parts, the verbal (SAT-V) and mathematics 
(SAT-M) sections. These two parts test understanding of verbal material 
and mathematics reasoning ability, respectively. 

TOEFL data were available from the five-part test (presumably 
obtained via International administrations]. 



Results 



For Group I , correlations between ACT subtest scores and CPA ranged 
from -.11 to .05 (N • 42). For Group 2, correlations between SAT and CPA 
were SAT-V: -.257 SAT-M: '32 (N - 26). For Group 3, the correlation 
between TOEFL and GPA was .05 ( N • 16). 

For the other variables, the most reliable correlations with GPA are 
based on the full sample of 126 students (Group 4). These correlations 
were high school average: .15; age -.17; and years out of school: -.01. 
Only the correlation involving age was significant at the .05 level. 

Multiple regression analyses produced many results, the most salient 
of which are summarised here. For Group 1, the multiple correlation 
involving all predictors with GPA was .47 (ns), and the multiple correla- 
tion with the best set of predictors (ACT English, ACT Natural Science, 
high school average, and age) was .46 (ns). The best predictor, high 
school average, correlated significantly with GPA (^ " .33), but the 
addition of other predictors via stepwise multiple regression resulted In 
nonsignificant multiple correlations. 

For Group 2, the multiple correlation involving all variables was a 
nonsignificant .59, and the multiple correlation with the best set of 
predictors (SAT-V, SAT^M, age, and years out of school) was a nonsignifi- 
cant ,58. The multiple correlation for the best two predictors (SAT-V and 
SAT-M) was a significant .49; addition of other predictors via stepwise 
multiple xegretSiOP .yielded nonsigniflCM jiprrelations , 

For Croup 3, the multiple correlation involving all variable? was a 
nonsignificant .64. The most effective prediction equation wss that 
involving age only (r - -.60). Stepwise regression showed that the 
multiple R for the two "bast predictors (age and years out of achool) was a 
significant .63; correlations involving additional variables were not 
significant. 
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For Group 4» the full unplc* t\m nultlplo correlation involving dl 
three variables'— high actwol average, age» and yeare out of achool—vaa a 

significant .26. The beet prediction equation was that involving all of 

these variables. 



Conclusions 



This BtwAy provides data that may be useful to adnissions officers. 
For the full sample of 126 subjects, undergraduate CPA was predicted by 
the combination of high school average, age, and years out of school. 
However, the relationship was low enough to suggest that use of equations 
involving these scores may have only minimal value for making admissions 
decisions. The TOEFL, SAT, and ACT were not good predictors of undex^ 
graduate GFA for ttese Iranian students, and it is recommended that these 
tests not be used in making admissions decisions for such students. 

Suggestions for further research include use of a larger sample and 
inclusion of sttidents from a wider geographic area and a wider variety of 
colleges. A departmental test in Farsi, the Iranian students' native 
language, a* well as tests in English, could be of value in assessing 
Iranian students* kn(»d'ledge as part of the admissions process. 
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6. Angelis, P. J. (1977). Language tettlng and Intelligence testing: 
Friends or foes? In J. E. Reddon (Ed.), Proceedings of the First 
International Conference on Frontiers in Language Proficiency 
and Doiainance Testing . Occasional Papers on Linguistics, No. 1. 
(^arbondale, IL: Southern Illinois University. (ERIC Document 
Reproduction Service No. ED 145 677) 



Purpose 



This paper revlewa data and issues pertaining to relationships 
between language proficiency and intelligence. Attention is given to the 
requirements of verbsl proficiency teat iteas ai^ tVm degree tb vhich such 
items tap intelligence rather than language skills alone for nonnative 
English speakers. Special attention Is given to TOEFL section scores and 
patterns of performance on these sections. 



Discussion 



The work of Angoff and Sharon (1971) suggests that native English 
speakers have little difficulty with TOEFL. However, an unpublished study 
with 88 native English-speaking high school students slu>ws that, while the 
subjects had little difficulty with TOEFL overall, the Structure and 
Written Expression and the Reading Comprehension and Vocabulary sections 
were found to contain items that were more difficult than would be expect- 
ed for native speakers. In the case of each of these subtests, one-fourth 
to one-fifth of the items wsre answered incorrectly by at least 80 percent 
of the subjects. Lack of grammatical skills influenced the difficulty of 
the Structure and Written Expression section. The difficulty of the 
vocabulary items was affected by abstractness and frequency of vocabulary. 
The difficulty of the reading comprehenstod items as influenced by the 
need to make complicated judgments and inferences. The occurrence of 
unexpectedly difficult iteas for native English speakers complicates the 
interpretation of performance on such items. 

Another unpublished study involved analysis of data from TOEFL 
(presumably the five-part version] and the Graduate Record Examinations 
(ORE) Aptitude Test given to 91 foreign students applying for «lmlssion to 
Texas A & M University. [See Summary No. 4, American Association. .., 
1971, for a brief description of tha ^«IE Aptitude Test.] The correlation 
between TOEFL score and GRE total sctre was found to be .33 (£ < .0001), 
which was slightly lower than the corre.Tatlon of .55 (£ < .001) between 
the TOEFL score and the GRE verbal subseore. A lower correlation of .31 
(£ < .01) was observed between the TOEFL score and the GRE quantitative 
score. The correlation between tte GRE verbal score and the GRE quantita- 
tive score was only .13 (ns). 
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When Che subjects were divided into those who scored high and those 
who scored low on the GRE verbal section and on TOEFL, 87 of the 91 
subjects scored low on the GRE verbal section, and these subjects were 
•bout equally divided between those with low and high TOEFL scores. 
However, when a similar subdivision of subjects was made using GRE total 
•core in place of GRE verbal score, the high GRE category was found to 

contain 36 subjects snd the low GRE category, 55 si&jects; within both the 
low GRE and high GRE groups, there were about equal numbers of low 
and high TOEFL scorers. Thus, addition of the GRE quantitative score 
apparently produced a more balanced array of scores. 

Comparison of the content of TOEFL and GRE items shows that GRE 
verbal items stress more difficult vocabulary, longer passages of text, 
and more complicated and abstract inference requirements. The GRE verbal 
test makes great demands on nonnative English speakers because of the 
combined intellectual and language skills needed to comprehend and work 
items. It is concluded that the GRE verbal test should not be interpreted 
as testing language proficiency, a purpose that is better fulfilled by 
TOEFL. 

Research done in Sweden found that scores on measures of inductive 
and logical reasoning in Swedish appeared to be associated with English 
reading rest scores. Also, the score on a measure of intellectual dellb^ 
erateness showed a negative correlation with the score on an English 
language test. These various results suggest the need for more research 
on the relation between language and intelligence. 
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7. Angeiis, P. J. (1979). TOEFL ir. recent years. In B. Spolsky (Ed.), 
Some major tests. Advauces In language testing se ries; I. 
Papers In applied linguistics. Arlington, VA: Center for 
Applied Linguistics. (Edited volume available as ERIC Document 
Reproduction Service No. ED 183 004) 



Purpose 



This is a descriptive paper dealing with developments in TOEFL since 
1973. lit complements the papers by Jameson & Malcolm (1973), Oiler & 
Spolsky (1979). and Palmer (r965). summarized in this collection, which 
describe the history of TOEFL prior to that time.) 



Discussion 



Before 1973, the College Entrance Examination Board (CEEB) and 
Educational Testing Service (ETS) held Joint responsibility for the TOEFL 
program. In July 1973, a new arrangement was formed, whereby responsibil- 
ity for the direction of the program was assumed by both CEEB and the 
Graduate Record Examinations Board (GREB), with ETS continuing to manage 
the program's operation. This change reflected the increase in the number 
of graduate applicants taking TOEFL. 

One outcome of this change was the replacement of the National 
Advisory Council on TOEFL with the TOEFL Policy Council as the policy- 
making body for the test. Whereas the earlier council was a general group 
representing several different constituencies, the new Policy Council had 
a more strictly defined structure. It r.onsisted of three members appointed 
by CEEB and three members appointed by GREB— who formed the six-member 
Executive Committee — along with nine other members representing foreign 
student advisers, sdmissions officers, government agencies, and teachers 
of English as a second language. 

The Committee of Examiners, which had existed since 1966, was no*' 
made a standing committee of the Policy Council. Its role was enlarged to 
Include regular review of test items and coniributlon of general advice on 
test content. A second standing committee, established in 1976, was the 
TOEFL Research Committee, a five-member panel formed to review research 
proposals and monitor research projects. 

The TOEFL program has continued to grow. As of 1977, there were five 
annual administrations of TOEFL, ud the number of special centers offer- 
ing the test monthly had grown to &bout '5. The number of TOEFL examinees 
had risen to nearly 150,000 in the year 1976-77. 

The test was revised as a result of data obtained in a study by Pike 
(1979). The English Structure and Writing /ibillty subtests were combined, 
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partly because of the high correlation found between them; the Seadlng 
Comprehension and Vocabulary subtests irere combined for similar reasons* 
The riew test thus consists of three sections rather than five, and the 
change In format resulted in a reduction In number of itaos and testing 
time as well as In changes In Item types* {See Introduction for descrip- 
tions of the three-section and five-section tests*] Since the equating 
system and score scale remain the same, total scores on the three^part 
test can be Interpreted In the same way as total scores on the old five-- 
part test (although section scores cannot be compared)* Since Listening 
Comprehension now contributes one-third of the score, ^ather than one- 
fifth as before, some shift in the role of listening may occur, and 
further data are needed to determine the nature and extent of such a 
shift. 

Certain assumptions remain the same as they were in 1963, when the 
test was first developed* Notably, a normative scale is still used, 
whereby an examinee's proficiency is expressed in relation to that of all 
others who have taken the test* 

Regarding the use and interpretation of TOEFL scores, surveys have 
indicated that most Institutions use section scores, particularly the 
Listening Comprehension score, in making admissions decisions* Also 
regarding score interpretation, it has repeatedly been stressed that TOEFL 
scores are not appropriate predictors of future grades* Admissions 
decisions presumably should be made by first examining a student's past 
academic record and then using TOEFL scores to help determine whether the 
student has the necessary English proficiency to do the required academic 
work* Yet TOEFL does provide predictive information of a sort* If the 
role of English proficiency in different programs of study were to be 
determined, guidelines could be established concerning the meaning of 
TOEFL scores for students at different levels and programs of study* 

Since the formal program of TOEFL research began in 1976 under the 
governance of the TOEFL Research Committee, several projects have been 
initiated. (The studies mentioned are now completed and are among the 
first reports published in the TOEFL Research Report series, which are 
summarized in this collection* ] 

A new version of the TOEFL manual, due to appear shortly as of the 
time of this writing, provides data on the three-part test. [The 1981 
manual, which postdates the 1976 version to which Angells refers. Is 
summarized in the present collection, with data presented for the revised 
test* ] 

An invitational conference in October 1977 was attended by 10 speci- 
alists in English as a second language* Discussions among these special^ 
lsts> the TOEFL Committee of Examiners, and ETS staff led to suggestions 
for further shifts in emphasis In the T01EFL— principally , to use QK>re 
extended contexts and more realistic situations, and to place more stress 
on academic contexts, since the test is used primarily for college and 
university !:uimissions decisions* 
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Am ce&tinued input is provided hf the various TOEFL eoudttess «nd 
other specialists in the field, and as research indicates new areas for 
investigation. It is appropriate that the TOEFL program be responsive to 
stsggestions for further change. 
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Purpose 

This study sought to determine the relationship betueen performance 
on TOEFL and performance on two couax>nly used meastires of verbal aptitude: 
tfao verbal section of the Graduate Record Examinations Aptitude Test 
(GRE-V), taken by prospective graduate students, and the verbal section of 
the College Board Scholastic Aptitude Test (SAT-V), taken by prospective 
undergraduates. Because SAT examinees also take th« Test of Standard 
Written English (TSWE), which is often used to place native English 
speaking students in freshman English courses, scores on the TSWE were 
compared with scores on the TOEFL also. 



Method 

A telephone survey of 50 large universities was first conducted, 
which showed that the TOEFL, the SAT, and the GRE are the most commonly 
required tc^sts of foreign applicants to undergraduate and graduate pro*> 
grams. Sublsequently , the experimental sample for the present study was 
selected, which consisted of 396 volunteers. The subjects, w^.o included 
210 undergraduate applicants and 186 graduate applicants, took either the 
GRE-V or the SAT-V and the TSWE after a regularly scheduled normlng 
administration of the three-section TOEFL. iThe SAT and T^ are briefly 
described in Summary No. 2, Aldermau 1982; the GRE Aptitude Test is 
briefly described in Summary No. 4, American Association..., 1971]. All 
testing was carried out at 13 domestic test centers. Background data 
indicated that the group was typical of the TOEFL population. Thirty-five 
different native languages were listed by the graduate students and 30 by 
the undergraduates. 

Mean scores produced by the sample were compared with mean scores for 
representative comparison groups of about 1,500 examinees each: (a) 
examinees ra^idomly selected from the total population of examinees who 
were administered the same form of TOEFL, (b) native speakers who took the 
same form of the GRE-V, and (c) native speakers administered the same 
forms of the SAT-V and TSWE. Performance on TOEFL was then correlated 
with, and compared to, performance on one of the other tests. 



Results and Conclusions 

Table 1 depicts basic descriptive statistics on the four measures 
included in this study for the experimental and comparison groups. The 
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ae«n score of 269 on the SAT-V achieved hy the nomuitive tmdergradisates 
shows that this group found the test very difficult. The aesn score of 
274 achieved by graduate nonnatives on the GRS-V was also well below the 
mean score for native speakers » even though the aean TOEFL score of 
graduates Included in this ssaple (523) was shove tte nean for all gradu- 
ate exaalnees on TOEFL (506)* Although scores on the SAT-V and GRE-V for 
both nonnatlve groups wre attenuated, tte undergraduates were not as far 
f 1.5 SD) below their native spealcing counterparts as were ttu! graduates (2 
SD)» It should also be noted that the reliability of both verbal aptitude 
Beaaures was conaiderably lower for nonnatlves than for natives. 



Table I 

Descriptive Statistics on Fotur Verbal Measures 
for Experimental and Comparison Groups 



Test and subgroup N >fean SD Reliability 



TOEFL 


( unde rgradua te s ) 


210 


502 


63 


.94 




(graduates) 


186 


523 


69 


.95 




(coDparlson group) 


1,540 


493 


75 


.95 


SAT-V 


(nonnatlves) 


210 


269 


67 


.77 




(natives) 


1,765 


425 


106 


.91 


TSWE 


(nonnatlves) 


210 


28 


8.8 


.84 




(natives) 


1.765 


42 


11.1 


.89 


6RE-V 


(nonnatlves) 


186 


274 


67 


.78 




(natives) 


1,495 


514 


128 


.94 



The natlve-nonnatlve performance differential on the TSWE was slight- 
ly less (1.4 SD) than was observed for the SAT-V, and the rexlabillty of 
the TSVE was slailar for both groups. 

It Is concluded that the SAT-V, TSWE, and GiiE-V are all difficult for 
nonnatlves. Because their scores cluster in the lower extreae of each 
test scale, interpretation of scores of nonnatlves on these tests can be 
coapllcated. 

The correlations aaong these Measures are depicted In Table 2. The 
correlation of .65 between TOEFL total and 6RE-V indicates that these two 
scores are related, altlwugh the skills being measured are not identical. 
The Listening Comprehension section of TOEFL shoi^d the lowest correlation 
with GRE-V, which is to be expected, since listening akill is not measured 



46 



with the GRE-V, The other two TOEFL sections exhibited stronger and 
nearly equal correlations with the GRE-V. The same i>attem occurred with 
the SAT-V. The TSWE correlated more strongly with the TOEFL than did 
either of the other two tests, with the weakest relationship between the 
TSVE and any specific TOEFL section involving Listening Comprehension 
(.51) and the strongest relationship involving Structure and Written 
Expression (.71). The latter finding offers support for the construct 
validity of the Structure and Written Expression section of TOEFL, 
since it is similar in format to the TSWE. 



Table 2 

Correlations between Scores on TOEFL and Scores 
on Three Comxnon Achaissions Tests ' 











TOEFL Section 












Listening 
Comprehension 


Structure and 
Written Expression 


Reading Comp. 
and Vocabulary 


Total 
TOEFL 


GRE-V 
SAT-V 
TSWE 


(N - 
(? - 
< N - 


186) 
210) 
210) 


.52 
.45 
• 51 


.61 
.64 
.71 


.62 
.68 
.66 


.65 
.68 
.72 
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9* Ango£f» W# H»» & Sharon, A. T« (1971)* A comparison of scores earned 
on the Test of English as a Foreign Language by native American 
college students and foreign applicants to U.S. colleges* TESOL 
Quarterly t 5^5 129-136# 

Purpose 



This study compared performance of native American students and 
foreign stvdents in performance on TOEFL. 



^tethod 



The native American subjects consisted of 71 entering freshmen at a 
western state university who averaged at the 29th percentile on the 
English subscore of the American College Test (ACT). The ACT English 
subtest is part of a battery of tests for American students applying to 
U.S. colleges and universities and measures kncwledge of English usage. 
The subjects were administered the five-part TOEFL in February 1969. The 
sample of foreign students consisted of all candidates given an operation^ 
al (presumably International] TOEFL over a three-year period from 1964 to 
1967 (N • 34,774). 



Results and Conclusions 



The average score of the native Americans was about tw standard 
deviations higher than that of the foreign students on Listening Compre- 
hension, English Structure, and Vocabulary and about one standard devia- 
tion higher on Reading Comprehension and Writing Ability. Furthermore, 
the American students* score distribution was narrcwr than that of the 
foreign students and was highly skewed in the negative direction, with a 
higher proportion of students earning maximum or near-maximum scores. 
These results show that the test was extremely easy for the native Ameri- 
can students. 

The correlation between ACT English and TOEFL score for the native 
American subjects was a relatively low .64, which cannot be attrilnited to 
unreliability of the tests (reliabilities of ACT English and TOEFL are 
reported by the tests' publishers to, be .88 and .97, respectively). Thus, 
TOEFL appears to measure somewhat different language skills from those 
measured by the ACT &iglish subtest. 

Analysis of item difficulties showed that only 17 of 130 Items i^re 
more difficult for the native American th^n for the foreign students. 
(This analysis excluded those items for which £ values, or prop%/i. tlons of 
the groups answering the items correctly, exceeded .93, since statistics 
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ott tuch ittam tend to b« unralUbU) • This r«iult further atccic* to the 
ease of TOEFL for native Aaericaa studeatt. Of theie 17 itma, 13 were 
froa the Writing Ability section, which tape exaBln«es' knowlwige of 
gravaatical foras. The relatively large nmber of such Iteas say reflect 
the fact that Aaericans are frequently exposed to Incorrect graaaatical 
fozas in colloquial English. 

Listening Coaprehension iteos were uniformly easier for the Aaerican 
than for the foreign students; this finding aay result froa the fact that 
English is transmitted in spoken fora aore often for the fomcr students 
than for the latter. English Structure iteas were also unifoxaXy easier 
for the Aaericaa students » which could be due to the fact that this 
section is intended to identify the types of language errors character- 
istic of foreign speakers of English. 

In general, the relative ease of TOEFL for the Aaerican students and 
the narrowness and skewedness of the score distribution for these students 
indicate that TOEFL is not an appropriate test of English for native 
Aaerican students. 
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10. Angoff, V. H. , & Sharon, A. T. (1974). The evaluation of differences 
in test perforaanee of two or aore groups. Educational and 
Psyclwloglcal Heasureaent , 34 , 807-816. 



Purpose 



This study sought to deteralne whether there are differences among 
language groups In the relative difficulties of various vocabulary it^s 
on TOEFL. It also served to denonstrate statistical procedures by which 
to address this question and identify itens that are particularly easy or 
particularly difficult for a given language group* 



Method 

The subjects were examinees taking *1the five-part TOEFL at a regular 
{presumably International] test administration in October 1969. All test 
candidates (a total of 6,120) in six native language groups iffere included: 
German, Spanish, Arabic, Chinese, Japanese, and Gujaratl. The study also 
included a sample of 1,000 cases drawn at rand(»B from the general popula- 
tion of examinees at X.\at s«aie test administration. Performance on the 40 
items in the Vocabulary section was examined. 



Results and Conclusions 

An analysis of varlTance was conducted with two factors; language 
group and specific vocabulary item. A significant interaction between 
items and groups showed that the patterns of performance on the 40 vocabu' 
lary items were different for the six language groups. Thus, ttus rank 
order of item difficulties tended to vary with the native language of the 
examinee. 

In additional analyses, the difficulty level of each of the 40 items 
for a given language group was plotted against the difficulty level for 
the general sample of 1,000 examinees. This was done for each of ttut six 
language groups, thus defining six elliptical scatterplots. For each 
language group, items falling outside this ellipse were identified as 
items that were particularly easy or particularly difficult for that 
group. Although the specific items so identified are not indicated in the 
report, distri).mtions of deviation are presented ss a i^wsns of demonstrat- 
ing the value of the statistical proc^ure. 

The study aakes no attempt to assess item differences in terms of 
linguistic considerations. It does suggest, however, that the itea» 
measure sufficiently different aspects of English vocabulary that they are 
not consistently aore difficult for one group than another. The study 
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«lto demottstratet a itatlBtlcal procedure that can be used to idtotlfy 
thoaa itema that are unuaually eaey or unusually difficult for a given 
language group. 
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11. Ayert, J. B., & Pttcrt, R, H. (1977). Predictive vaUdlty of tl» 
Test of BAgXlsh e Foreign Language for Asian gr«luate stu- 
dents in engineering* chemistrjri or Mtheaatice. Educational 
and Psychological Measurement , 37 , 461-463. 

Puri^se 

The relationships aaong college grades, Graduate Record Examinations 
(GRE) Aptitude Test scores* and TOEFL scores «ere exaained for 50 Asian 
students who had completed master's level programs in engineering, 
chemistry, or asthematics. 



Method 

The subjects were 50 male foreign studeats enrolled at Tennessee 
Technological University. The subjects* countries of origin were Asian 
1 identified simply as "Republic of China, India, Thailand, etc."]. All 
subjects had taken TOEFL prior to admission to the university. Fifteen of 
the 50 students had completed the GRE Aptitude Test, which yielded a 
verbal score (GRE-V) and a quantitative score (GRE-Q). [The GRE is 
briefly described in Summary No. 4, American Association. *• , 1971.] 
Overall grade-point average (GFA) for each subject was obtained frc» 
university records. 



Results 

Table 1 presents the principal data of the study. A stepwise 
regression analysis was used to predict GPA from TOEFL and GRE-V scores. 
The resulting equation was GPA - 0.004 TOEFL - 0.002 GRE-V "f- 2.38. The 
multiple R for the equation was .71 Xz < -05, dt - 12). Addition of GRE-Q 
to the equation failed to increase the multiple R. 

/ Table 1 

Descriptive Statistics and Intercorrelations among Variables 





N 


Mean 


SD 


Correlations 










GPA 


TOEFL 


GRE-V 


GPA 


50 


3.61 


0.31 








TOEFL 


50 


491 


51.30 


.40** 








15 


286 


73.50 


.22 


.76** 






15 


645 


63.70 


.55* 


.64** 


.47 


*Signlficant at the ,05 level. 
**Sigm,iieant at the .01 level. 
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Conclusloiui 



Th« results suggest thst TOEFt asy be s useful predictor of Aslsn 
students' perforasnce In asster*s level progrsas In engineering, cheais- 
try» or astiisastics. A coabinstion of TOEFL sod GRE-V scores sppesred to 
be a reaeonable predictor of success* While It nay not be possible to 
generalise to other acadeaic areas or to other ssaples of graduate stu- 
dents, the results suggest the predictive value of TOEFL in selected 
areas* 
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12. Blanchard, J. D. , & Heedy, R. (1970» September). The relationship of 
a test of English as a gecond language to laeasurea of achieve- 



ment and self-concept in a Baaple of Arorican Indian students * 
Recearch and Evaluation Report Series No. 58. Bureau of Indian 
Affairs, U.S. Department of Interior. (Reprinted 1977). Paper 
presented at the meeting of tl^ American Psychological Associa- 
tion, Miami Beach, FL. (ERIC Document Reproduction Service No. 
ED 147 090) 



The objective of this study was to understand factors contributing to 
the poor achievement levels of American Indian students. The study 
examined interrelationships among scores on tests of English language 
proficiency (including TOEFL), e4|>cational achievement, and self-concept. 
One hypothesis was that primary language and self-identity may 'be closely 
connected for American Indian students. 



The 99 subjects consisted of 49 males and 50 females in the eleventh 
and twelfth grades at the Albuquerque Indian Sclwol, an off-reservation 
boarding school managed by the Bureau of Indian Affairs. Six Indian 
language families, or tribal groups, were represented. Navajos comprised 
the largest group of students at the school (80 percent) and in the sample 
(N - 69 of 99) . 

The tests administered included the five-part TOEFL, the Iowa Test of 
Educational Development (ITED), the Tennessee Self-Concept Scale (TSC), 
and the Southwestern Indian Adolescent Self-Concept Scale (Q-sort). The 
ITED is an achievement test consisting of eight subtests: (a) understand- 
ing basic social concepts, (b) general background in the natural sciences, 
(c) correctness and appropriateness of expression, (d) ability to do 
quantitative thinking, (e) ability to Interpret reading materials in the 
social studies, (f) ability to inte pret reading materials in the natural 
sciences, (g) ability to interpret literary materials, and (h) general 
vocabulary. The TSC consists of self-descriptive statements the subject 
uses to portray himself or herself. Scores are derived on eight scales: 
(a) personal self, (b) family self, (c) social self, (d) total P score 
(i.e., sense of self-worth), (e) defensiveness, (£) general maladjustment, 
(g) personality disorder, and (h) personality integration. The Q-sort 
test measures how adolescents feel toward themselves. All tests except 
one were administered under classroom-like conditions by teachers, 
with some aid from a school psychologist. 



Purpose 



Method 
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Results and Coviclusions 

One-way analysis of variance was used to determine whether tribe, 
sex, and grade were associated with differences in individual test scores. 
Four of the five TOEFL, subscores and the total score were found to be 
significantly higher for Apaches than for other tribes. Scores on two 
subtests of the ITED (understanding of basic social concepts and general 
background in the natural sciences) were significantly higher for twelfth 
graders than for eleventh graders. TOEFL subscores did not differ signi- 
ficantly across grades, except that scores on the Writing Ability subtest 
were higher for twelfth graders. Only one significant difference between 
grades wat> found for scales on the TSC and Q-sort test. Sex and tribe did 
not have significant effects on scores on the ITED test, Q-sort test, or 
the TSC test. 

Multiple correlations were computed to investigate the association 
of all scores on the four instruments to grade, sex, age, and tribe 
separately. The results showed significant multiple Fls in every case. 
When the original zero-order correlations were examined, the following 
were significant. Grade correlated positively with ITED subtests in basic 
social concepts, background in the natural sciences, and interpreting 
materials in the natural sciences. Sex correlated positively with ITED 
correctness and appropriateness of expression and TOEFL English Structure, 
and negatively with Q-sort. Age correlated positively with ITED under- 
standing of basic social concepts and with TSC family self and general 
maladjustment. Tribe correlated negatively with TSE family self score. 

Inspection of TOEFL subtest and total scores revealed that the 
subjects had average scores falling in the range 300-449 on the Vocabu- 
lary, Reading Comprehension, and Writing Ability subtests. Listening 
Comprehension and English Structure scores were higher, more closely 
approximating the mean for foreign students as a whole, but the mean total 
TOEFL score of these Indian students was far below that of foreign stu- 
dents as a whole. 

The average ITED scores of subjects in this sample as well as stu- 
dents in the tenth grade at the Albuquerque Indian School were all below 
the 10th percentile nationally. The results for TOEFL and the ITED 
indicate that the subjects were not prepared for an English-speaking 
academic environment. Scores on a scale of the TSC that reflected sub- 
jects' self-image indicated that the subjects manifested low overall 
self-esteem in relation to the published norms for this scale. These 
Indian students also scored low relative to published norms on two other 
scales of the TSC test — the general maladjustment scale and the person- 
ality disorder scale. Scores on the Q-sort test, however, did not suggest 
pathological adjustment of Indians. The fact that the constructs under 
measurement by the TSC test were not normed or developed for American 
Indians must be considered in interpreting these findings. 

The results of the study demonstrated that the American Indians under 
investigation showed low skills in EngiJlsh as a second language. These 
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indlviduaXs scored Icwer than national norms on all of the tests used In 
this study. Perhaps American Indian students' cultural and linguistic 
backgrounds are at odds with the type of education provided In mainstream 
U.S. classrooms* 
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13* Bostlc, M, (1981). A correlational study of academic achiev«aent 
and the Test of English as a becond [sic] Language (TOEFL) 
(Doctoral dissertation, East Texas State University, 1981) • 
Dissertation Abstracts International , A3 , 468A. (University 
Microfilms No. 8116851) 



Purpose 

This research investigated the predictive validity of TOEFL for 
freshman foreign students enrolled at Southeastern Oklahoma State Univer- 
sity* 



Background 

Among the goals of the study was to determine the roles played by 
each of two admissions policies for foreign students. Under one policy, 
foreign students were admitted upon completion of a series of ESL classes, 
under a second admissions policy, foreign students were admitted if 
they scored at or above 460 on TOEFL. The study had four hypotheses. 
Hypothesis 1 was that there would be no difference in grade-point average 
(GPA) as a function of admissions policy. Hypothesis 2 was that there 
would not be a significant positive correlation between TOEFL score and 
GPA. Hypotheses 3 and 4 were the same as Hypothesis 2 except that GPA in 
nonverbal courses and GPA in verbal courses, respectively, were used as 
the criterion variables. Nonverbal courses were defined as those courses 
in which much of the subject matter could be understood without English 
language skills. Courses meeting this criterion included all courses in 
mathematics, physical sciences, computer science, drafting, engineering, 
electronics, chemistry, art, music, and physical education. Verbal 
courses were defined as courses that placed an emphasis on comnRinication 
of material through language. Such courses Included offerings in the 
fields of languages, humanities, social sciences, business, biology, and 
conservation. 



Method 



The 154 students studied represented 19 countries of origin. All 
subjects were enrolled as freshmen during the periods 1978-^79 or 1979^80. 
One hundred fifteen of the stiKlents for whom TOEFL scores were not avail^ 
able were admitted to academic study through completion of prescribed 
courses in English as a second language (ESL); 30 St^udents were admitted 
by achieving scores of 460 or better on the three-part TOEFL [presumably 
obtained via International or Special Center administrations]; ,ind nine 
students who scored below 460 on the TOEFL were admitted after cou -dieting 
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required ESL courses. Students admitted in fall 1978 and spring 19/9 were 
not required to present TOEFL scores. These students were required to 
have earned passing grades in 12 hours of ESL courses before they could 
enroll freely in non-ESL courses. Students admitted in fall 1979 and 
spring 1980 had to present TOEFL scores of 500 or above to gain adm^.ssion. 
In the 1979-80 school year, it was not possible to apply this TOEFL 
criterion uniformly; some students who had applied earlier fell under the 
1978-79 admissions policy, while other stiKients scored below 500 but were 
admitted nonetheless. 



Results 

In the case of Hypothesis 1, a t-test revealed no differences between 
the mean first-year college CPAs of subjects admitted under the two 
different admissions policies. The mean GPA for those admitted under the 
1978-79 policy was 2.81 versus a mean GPA of 2.67 for those ar^mitted under 
the 1979-80 admissions policy. 

Relevant to Hypothesis 2, a correlation of ,17 (jl - 39) was found 
between TOEFL score and freshman GPA; this correlation was not statisti- 
cally significant from zero. Hypothesis 3 was rejected, as a statisti- 
cally significant correlation of .50 (£ < .05, N - 39) was found between 
TOEFL score and GPA in nonverbal courses. Hypothesis A was not rejected, 
as a nonsignificant correlation of -.08 (N - 39) was found between TOEFL 
score and GPA in verbal courses. Hypotheses 2, 3, and 4 were re-examined 
using only those persons who scored at or above A60 on TOEFL. Re-examina- 
tion of Hypothesis 2 led to its rejection; a correlation of .33 (£ < .05, 
N - 30) was found between TOEFL score and freshman GPA. Consistent with 
the original findings. Hypothesis 3 was rejected and Hypothesis 4 was not 
rejected among subjects scoring at or above 460 on TOEFL (i.e., there was 
a significant correlation between TOEFL and GPA in nonveit but not 
verbal courses). 



Conclusions 



The findings support the conclusion that the two admissions cri- 
teria — completion of ESL course work and demonstration of a certain TOEFL 
score — were associated with the same level of freshman GPA achievement of 
foreign students. The analyses relevant to Hypothesis 3 show that TOEFL 
scores might be helpful in predicting foreign students' academic achieve- 
ment in areas of study that do not emphasize verbal skills. Failure to 
find a significant relationship between TOEFL score and GPA in verbal 
courses was inconsistent with findings of previous research. In general, 
it is concluded that TOEFL scores are a convenient and desirable method 
for screening foreign students for admission. 
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14. Campos'"Arcla» M*t & McVay, J. (1978, October). Graduate foreign 
s tudent admissions decision-making: An application of the JAN 

Technique . Paper presented at the meeting of the Southern 
Association for Institutional Research, Nashville, TN. (ERIC 
Document Reproduction Service No. ED 163 883) 



Purpose 

In determining whether to admit a foreign student to graduate school, 
several factors may be considered, such as undergraduate grades, standard- 
ized test scores, and so forth. The present study sought to determine 
the degree of commonality among faculty members with regard to factors 
used in admissions decisions for foreign graduate applicants. 



Method 



First, interviews were conducted with graduate administrators or 
department heads in academic departments at North Carolina State Univer- 
sity (NCSU) that had enrollments of 10 or more foreign graduate students. 
Based on these interviews, four variables were selected as most Important 
in admission of foreign graduate students: undergraduate academic record, 
TOEFL score [presumably obtained via International or Special Center 
administrations]. Graduate Record Examinations Aptitude Test quantitative 
score (GRE-Q), and letters of recommendation. (The GRE Aptitude Test is 
briefly described in Summary No. 4, American Association. , 197Kj 

Profiles were then constructed that simulated 40 hypothetical foreign 
graduate applicants, using various values on each of the four variables. 
Sets of these 40 profiles were sent to each of 85 faculty members who were 
asked to rate each hypothetical applicant on a five-point scale from 
"pooi" to "superior.** The raters li^re persons who had responsibility for 
admissions decisions and represented 16 graduate departments at NCSU that 
had enrollments of 10 or more graduate foreign students. Usable data were 
obtained from 52 respondents. 

The data were analyzed by means of the Judgment Analysis Technique 
(JAN), which uses a multiple linear regression approach to describe the 
rating policy used by each one of a group of judges. A Judge's policy can 
be described by the following type of multiple regression equation 



Where Y is the judge's rating, the Xs are the variables on which the 
rating is based and the Bs are the beta weights, which indicate the 
relative importance of each of the M variables in the Judge's decision. 
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B is a coasCant, and E is an error term. By analysing a Judge's ratings 
f8r the 40 hypothetical applicants, it is possible to determine the beta 
weights, or importance, ascribed to each of the four variables by each 
rater. 



Kesults and Conclusions 



The data show a considerable amount of difference among Judges in 
rating policies. At one extreme, one Judge assigned substantial weight to 
only one variable (GRE-Q), whereas at the other extreme, some Jwiges 
assigned substantial weights to all four variables. The many different 
rating policies used seemed generally to comprise seven different models, 
representing different combinations of the four variables: (a) GRE-Q 
only, (b) CPA and GRE-Q, (c) GRE-Q and letter of recommendation, (d) CPA 
and letter, (e) GPA, GRE-Q, and letter, (f) CPA, GRE-Q, and TOEFL, and (g) 
all four variables. (Within these seven models, variation among Judges in 
relative Importance of variables v&s also apparent.) 

Different rating policies among Judges within departments were also 
apparent, as shown in analyses of data from the 13 departments that were 
represented by two or more Judges. Similarly, when Judges were grouped 
according to years of experience in making admissions decisions for 
foreign graduate applicants (0-3 years, 4 to 9 years, 10 or more years), 
differences in rating policies were found within these subgroups. In 
general, then, no single policy regarding admission of foreign graduate 
students was found, even within departments or within subgroups of Judges 
with similar amounts of experience. 
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15. Chal» S. H. » & Woehlke, P. L. ( 1979, September). The predictive 
ability of standardized tests of English as a foreign language. 
In R. Silverstein (Ed.), Proceedings of the Third Intemationel 
Conference on Frontiers in Language Proficiency and Dominance 
Testing I Occasional Papers on Linguistics, No. 6. Carbondale, 
thl southern Illinois University. (ERIC Document Reproduction 
Service No. ED 186 476) 



Purpose 



This review paper considers issues and evidence regarding the use 
of tests of English as a foreign language as predictors of foreign stu" 
dents' acadesaic achievement. 



Discussion 



An important consideration in the use of a standardized test of 
language proficiency is its validity — content validity, construct valid- 
ity, and predictive or criterion validity. Also important is the reli- 
ability of a test. While English language proficiency tests are not 
typically constructed to reflect an English language learning theory, they 
nonetheless are expected to inform college admissions decisions for 
foreign students. 

The central issue in the admissions context is how useful proficiency 
test scores are in aiding admissions decisions. Relevant to this issue is 
the ability of proficiency test scores to predict certain acadoaic criter- 
ia — principally grade-point average (CPA). A review of selected studies 
aids in evaluating the utility of various English proficiency tests as 
predictors of college grades. 

Five studies evaluating TOEFL alone as a predictor of college grades 
and college final term *ating are cited. Validity coefficients in these 
studies ranged from .17 to .43, indicating that, at most, 18.5 percent of 
the variation in academic performance measures was predictable from TOEFL 
scores. When several predictors were used, such as combinations of TOEFL 
total and subtest scores or TOEFL scores and oral Interview ratings, 
multiple correlation coefficients ranging from .49 to .64 were observed. 
Three studies are cited that used TOEFL and the Graduate Record Examina- 
tions (GRE) Aptitude Test as predictors of GPA. (See Summary No. 4, 
American Association..., 1971, for a brief description of the GRE Aptitude 
Test]. These studies found a moderate level of predictability, as corre- 
lations ranged from .32 to .71. 

Three studies are mentioned that examined the predictive validity 
of the Michigan Test of English Language Proficiency (MTELP). [See 
Summary No. 1, Abadri, 1976, for a brief description of the MTELP.] The 
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criterlon neasures In these studies were CPA or grades in selected course 
areas. Sample siies ranged from less than 10 to 213. Validity coeffi- 
cients ranged from .28 to .52. The largest proportion of MTELP variance 
accounted for was 27 percent. 

Four studies reviewed used a combination of TOEFLj the MTELP, and 
other tests of English proficiency to predict college grades. Validity 
coefficients ranged from -.05 to .70 in these studies, with sample sizes 
ranging froa 17 to 402. The largest validity coefficients tended to occur 
for studies involving the fewest subjects. 

Three other studies involved prediction of college grades, college 
transcript ratings, earned credits, or an unspecified criterion of aca- 
demic success. These studies utilized locally developed proficiency tests 
or other measures of verbal skills or verbal proficiency as predictor 
variables. Validity coefficients in these studies ranged from .02 to 
.84. 

It is concluded that scores of tests of English as a foreign language 
(£FL) do not show a consistent and substantial relationship to academic 
achievement. Various factors might have attenuated the relationships 
reported. Unreliability is probably not a significant factor for the most 
prominently used tests. Variation in grading standarc's is a factor that 
may limit prediction of academic achievement. Also, the fact that the 
samples in these studies consisted of students who were admitted, and thus 
had a restricted range of English proficiency scores, could be a factor. 
Finally, there is the possibility that EFL tests are not measuring lan- 
guage and communication skills that are important for academic function- 
ing. 

College entrance examination test scores have shown higher relation- 
ships to college grades for native speakers of English than have English 
proficiency test scores for foreign students. Academic achievement in the 
student's native language and mathematical aptitude may need further 
investigation as predictors of a foreign student's academic achievement. 
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16. Charrow, V. R. , & Fletcher, J. D. (1974), English as the second 
language of deaf children. Developmental Psychology , 10, 
463-^70. 



Purpose 



It was theorized that deaf children of deaf parents learn English as 
if it were a second language, since such children usually learn sign 
language first whereas deaf children of hearing parents do not. Three 
derivative hypotheses were tested: (a) deaf children of deaf parents 
should perform better than deaf children of hearing parents on any test of 
language skills, (b) performance of foreign students on a test of English 
as a second language should resemble that of deaf children of deaf parents 
more than that of deaf children of hearing parents, and (c) the relation 
between a standard test of English skills and a test of English as a 
second language should be lower for deaf children of deaf parents than for 
deaf children of hearing parents. 



Method 



Twenty-six adolescents in a state school for Jthe deaf served as 
subjects; these included 13 deaf children of deaf^ parents and 13 deaf 
children of hearing parents. All were xiogenitally deaf except three 
children of hearing parents, who wre deaf by 18 months of age. The n«an 
age of the subjects with hearing parents (18.3 years) was significantly 
higher than that of the subjects with deaf parents (17.2 years). The 
groups were similar in socio-economic status. TOEFL scores of a sample of 
495 hearing foreign students were also available [the soiurce of these 
students is not specified]. 

The five-part TOEFL was administered, with the Listening Comprehen- 
sion section omitted. All references to English as a foreign language 
were deleted. The test instructions were given in Signed English, and 
sample questions were written on a blackboard, signed, and f ingerspelled. 

Grade placement scores from the Paragraph Meaning and Language 
subtests of the Stanford Achievement Test were also available for the 26 
deaf subjects. The Paragraph Meaning subtest measures the subject's 
ability to comprehend and draw inferences from written discourse, and the 
Language subtest taps the subject's proficiency in usage, punctuation, 
capitalization, dictionary skills, and sentence sense. 

Results and Conclusions 

The scores of deaf subjects with deaf parents were significantly 
higher than the scores of those with hearing parents for each of the four 
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TOEFL lubtesti (except Reading Coaprehension) , es well as for the total of 
the four aubteats and for hoth of the Stanford aubteata. Correlation and 
regression analyses also showed a relation between parentage (as a dichot- 
omoua variable) and test scores. These results support the first hypoth- 
esis, that deaf children of deaf parents outperform those with hearing 
parents. 

To test the second hypothesis, the nuaber of persons answering each 
item correctly was determined for each of three groups: foreign students, 
deaZ subjects with hearing parents, and deaf subjects with deaf parents. 
(Data were also obtained for the coabination of the two deaf groups.) 
Correlations were cooputed between pairs of groups, with individual items 
as the units of analysis; this was done separately for each of the four 
subtests of TOEFL. Correlations between the foreign-student group and 
each of the two deaf groups were of particular interest. These correla- 
tions were higher for subjects with deaf parents than for those with 
hearing parents, for each of the TOEFL aubtests. This result tends to 
support the second hypothesis in showing that, of the two groups of deaf 
subjects, those with deaf parents performed more similarly to foreign 
students on a test of English as a second language. (Similar results were 
obtained in a second correlational analysis, in which the dependent 
variable was the number of responses to the most likely wrong answer, 
rather than number of correct answers.) 

The second hypothesis, however, would also predict that correlations 
between the two deaf groups shoidd be lower than the correlation between 
foreign students and subjects with deaf parents. Such a pattern was 
obtained for the English Structure and Writing Ability subtests but not 
for the Vocabulary and Reading Comprehension subtests. Thus, evidence for 
the second hypothesis was mixed. 

The third hypothesis was that the correlations between TOEFL and the 
Stanford tests would be higher for deaf subjects with hearing parents than 
for those with deaf parents. Evidence for this hypothesis was equivocal, 
as correlations of the TOEFL subscores with the Stanford Language subtest 
were generally in the predicted direction, while correlations with the 
Stanford Paragraph Meaning subtest were not. 

In light of the mixed results for the second and third hypotheses, it 
is possible that deaf children learn only some aspects of English as if it 
wre a second langu£^e. In any case, however, the superior performance of 
the deaf children of deaf parents may result from having learned sign 
language at an early age. 
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17. Chase, C. I. , & Stallings, W. M, (1966). Testa of English luguage 
as predictors of success for foreign students . Indiana Studies 
In Prediction No. 8. Monograph of the Bureau of Educational 
Studies and Testing. Blooalngton, IN: Bureau of Educational 
Studies and Testing, Indiana University. 



Purpose 



This study assessed the relation of students' grade-point average 
(GPA) to TOEFL and other tests of English language proficiency for a 
sample of foreign students at Indiana University. The few studies done 
prior to this study suggest a low relation between English test scores and 
GPA, possibly because such tests measure differences In class cultture 
rather than Intellectual i^illty. 



Method 



The subjects were foreign greduate, undergraduate, and nondegree 
students attending Indiana University. 

The five-part TOEFL was one of three English tests used. (Scores 
presumably were available from International administrations.] There were 
37 students for whom both a TOEFL score and a GPA were available. Bell- 
abilities (KR-ZO) of the TOEFL subtests are reported to range from .64 to 
.94, iThese figures, presented in the Method section, appear to be 
reliabilities obtained from previous publications; reliabilities for the 
other tests used in this study are not reported. 1 

Also administered were two subtests of a test developed by Robert 
Lado at the University of Michigan. Lado Test B is a test of aural 
comprehension. Lado Test C consists of three parts: (a) Structure, a 
multiple-choice test requiring interpretation of English sentences; (b) 
Pronunciation, requiring distinction of letter soimds and knowledge of 
correct syllable stress patterns; and (c) Vocabulary, requiring selection 
of the words or phrases most like those underlined in a series of sen- 
tences. Lado test scores and GPA were available for 343 subjects; both 
Lado test scores and TOEFL scores were available for 50 subjects. 

A third test, termed the Pennstate test, was developed at Pennsyl- 
vania State University and has six parts: (a) in Soimd Dlscriainatlon, 
the subject sees a pair of words and circles the word that corresponds to 
the word being read; (b) in Incomplete Sentences, the sxibject completes a 
sentence; (c) in Word Fluency, the subject lists as many words as possible 
beginning with a certain letter; (d) in Reading Comprehension, the subject 
indicates tte waning of a paragraph via multiple-choice responses; (e) in 
the Scrambled Test, the subject constructs a sentence from words out of 
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order; and (f) In the Vocabulary Synonyms test, the subject selects the 
word most like a given stimulus word. This test was administered to 52 



subjects. 



Resultt, and Conclusions 



The two Lado tests correlated .67 with each other, suggesting a 
moderately high degree of overlap In skills measured by then. Correla- 
tions with first- and second-semester GPA, respectively, were; for Lado 
Test B. .28 and .22; for Lado Test C, .20 and .23. These correlations 
fc«re all significant at the .01 level (N - 343) . 

When students were grouped according to various categories, such as 
sex, sponsored vs. nonsponsored, and GPA above vs. GPA below a certain 
level, the correlations between Lado tests and GPA generally did not 
differ between groups. Exceptions were significantly higher correlations 
between Lado Test B and second-semester GPA for undergraduates than for 
(a) graduates and (b) contract (nondegree) students, perhaps because of 
the homogeneity of ability of the latter two groups. Analyses by lan- 
guage/culture groups and by major-field groups showed only a few cases in 
which correlations between the Lado tests and either first- or second- 
semester GPA were significant at the .05 level. These Involved the 
romance languages and Chinese of 12 language/ culture groups, and education 
and the humanities of five major-field groups. 

The five TOEFL subtests— Listening Comprehension, English Structure, 
Vocabulary, Reading Comprehension, and Writing Ability — correlated .22, 
.26, .23. -.07. and .14, respectively, with GPA [whether first-semester, 
second-semester, or combined GPA is not indicated]. The correlation of 
first-semester GPA and Lado Test B was cwapared statistically with the 
correlation of GPA and TOEFL Listening Comprehension, sincts these two 
tests were comparable in format; for similar reasons the correlation of 
first-semester GPA with Lado Test C was compared with the correlation of 
GPA with TOEFL English Strxicture. These differences were not significant. 

The two Lado tests correlated significantly with each of the TOEFL 
subtests. Although correlations with the Lado tests «re somewhat lower 
for TOEFL Listening Comprehension and Reading Comprehension U's «■ .45 to 
.59) than for the other TOEFL subtests (.58 to .87), in general th&re 
appeared to be substantial commonality in skills measured by these tests. 

The Pennstate tests failed to correlate significantly with GPA 
(r's - -.15 to -f.lS). 

In general, while the Lado tests correlated significantly with GPA, 
based on an N of 343, these correlations were of approximately the same 
magnitude as the nonsignificant correlations between TOEFL and GPA, based 
on an N of 37. Given these results, and given the moderate to high 
correlations between the Lado tests and TOEFL, it appears that the Lado 
tests and TOEFL measure quite similar characteristics. 
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None of the teiti used here appeared to be very tuccessfuX in pre- 
dicting students* acadenic performance. Interpretation of this result is 
rendered difficult by the fact that CPA may be less reliable for foreign 
students than for domstic students; also, subjects who took TOEFL and the 
Pennstate test may have been less represents £ive of the total foreign 
student population at Indiana University than the larger sample who took 
the Lado tests. Nevertheless, the data suggest that better prediction of 
academic success for foreign students may be achieved by examining vari- 
ables other than English language ability. 
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18. Clark, J. L. D. (1977). The performance of native speakers of 
English on the Test of English as a Foreign Language (TOEFL 
Research Rep. No. 1). Princeton, NJ: Educational Testing 
Service. 



Purpose 

This study examined the performance of native English speakers on 
TOEFL and sought to Identify characteristics of items that are difficult 
for these examinees. 



Method 

The subjects were 88 college-bound seniors from two New Jersey high 
schools, who were paid for their voluntary participation. All participants 
were native speakers of English and had had little or no exposure to 
languages other than English except in school foreign language courses. 

The three-part TOEFL wan administered one week prior to graduation in 
each of the two schools. Two alternate forms of the test were adminis- 
tered, each randomly assigned to approximately half the students per 
school. The test was described only as an experimental English achieve- 
ment test. The instructions were very similar to those used in the 
administration of an operational TOEFL. In addition, however, the in- 
structions urged the subjects to "make special note of any individual 
questions which seem substantially more difficult than the others." For 
the Structure and Written Expression section and the Reading Comprehension 
and Vocabulary section, the test administrator sought to provide suf- 
ficient but not excessive time. The times acttclly allotted In these 
sections were about 80 percent of those allotted in operational test 
administrations with foreign stxuients, and this amount of time proved to 
be adequate for these native English speakers. 

After completion of the test, each subject completed a short ques- 
tionnaire asking for judgments as to the relative difficulty of each 
section and subsection of the test. 

Results and Conclusions 

The subjects scored high on the test, with mean scores of 134.42 (SD 
• 10.19) and 134.91 (SD - 11.44) of a possible 150 on the two forms. The 
range of scores was very restricted and the distribution was highly skewed 
in the negative direction. These results contrast with considerably lower 
scores, and wider, nonskewed score distributions for the foreign-student 
group on which these forms were initially scaled. These results indicate 
that the test would not be appropriate as an Instrument for differentiat- 
ing among native English-speaking students. 
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To compare the different test sections, a "percent^fail" rate v»s 
computed for each test section; this was the percentage of subjects 
incorrectly answering or omitting an item, averaged across items in a 
section. These rates were 4.4, 14.6, and 12.1 for the three sections, 
respectively, showing that the Listening Comprehension section was easier 
than the other two sections* The subjects were also asked to rank order 
the sections in difficulty. The subjects gave lower average rankings to 
the Listening Comprehension section than to the other tw, showing t. at 
they perceived this section as the least difficult* 

Percent-fall rates and difficulty ratings were also obtained for each 
part of each section. Within Listening Comprehension, subjective diffi- 
culty ratings indicated that the subjects found the third part, listening 
to passages, more difficult than either of the other parts, listening to 
short statements or dialogues, perhaps because the passages require more 
concentrate'' attention. Nevertheless, performance all three parts, of 
Listenlni? . . n i Uon was relatively high. 

Wit^ ructure and Written Expression section, percent-fail 

rates showta ..^ error recognition itens v^re considerably more difficult 
than sentence completion items, althotigh the subjects* difficulty ratings 
did not show a difference in perceived difficulty between these two item 
types. 

Within the Reading Comprehension and Vocabulary section, percent-*f ail 
rates showed the vocabulary items to be considerably easier than the 
reading passages, a difference that was also reflected in the subjects' 
difficulty Judgments. 

In an analysis of Individual items, tho. e items with percent-fail 
rates above 20 percent were noted. Taking both test forms together, there 
were only three such items in the Listening Comprehension section, but 22 
in Structure and Written Expression ^^nd 22 in Reading Comprehension and 
Vocabulary. Most of the difficult Items for Structure and Written Expres- 
sion Involved structural aspects with which college-bound native English 
speakers should be familiar; thus, the data do not indicate the presence 
of an appreciable number of "faulty" grammar items. For the eight diffi- 
cult vocabulary items, a clear relationship to lexical rarity or other 
obvious factors could not be discerned. The 14 difficult Reading Compre- 
hension items were primarily ones requiring the subject to summarize or 
interpret the point of a passage; however, some factual questions were 
also answered incorrectly. 

In general, the data suggest that TO'Fl not an appropriate 
instrument for differentiating among native Euglish speakers with respect 
to English prcficlency. Assessment of native English speakers* perform- 
ance might be useful in development of future tests. However, elimina- 
tion of all items found difficult for this population would not be appro- 
priate, because such a procedure would exclude items that test important 
components of English proficiency. 
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19. Clark, J, L. D, , & Swlnton. S. S. (1979). An exploration of speaking 
proficiency measures in the TOEFL context ^ TOI^rL Research Rep. 
No. 4; ETS Research Rep. No. 79-8). Princeton, NJ: Educational 
Testing Service. (ERIC Document Reproduction Service No. ED 201 
641) 



Purpose 

The pturpose of this sttdy was to develop and field test a semi-direct 
measure of oral English skills that could be administered within the 
context of the TOEFL program. 



Method 



The main sample conJ:ained a total of 155 foreign students enrolled in 
English as a second language classes at the University of Florida or at 
the University of Pennsylvania during the fall of 1978. 

Eleven experimental item types were selected for inclusion in an 
experimental test of oral English skills that was administered on tape 
with the aid of a picture booklet. The item types included (a) autobio- 
graphical questions, (b) use of a pictured noun in a sentence, (c) use of 
a pictured verb in a sentence, (d) reading aloud, (e) sentence repetition, 
(f) dehydrated sentences, (g) f ill-in-the-blank. (h) telling a story based 
on a sequence of pictures, (i) answering multiple questions based on a 
single picture, (j) a simulated telephone conversation with a voice on a 
tape, and (k) persuasive speech based on a sequence of pictures. 

Parallel forms of the experimental test were given to 12 examinees, 
along with a long and short Foreign Service Institute (FSI) type oral 
proficiency interview and the three-part TOEFL. Based on the results, two 
item types, persuasive speech and use of a pictured verb in a sentence, 
were eliminated and a new item type, sentence completion, was added to a 
developmental edition of the test. Two forms cf the test were constructed, 
each containing 97 items, and both forms were administered to the main 
sample of 155 students. The subjects also took an FSI-type interview and 
the three-part TOEFL. FSI raters indicated which of five ratings the 
examinees received on the five basic criteria: prcnounciation, grammar, 
vocabulary, fluency, and comprehension. Items on the developmental 
edition of the speaking test were scored for pronounciation, fluency, and 
gramoiar. 

A covariance matrix was developed relating all of the above vari- 
ables. This matrix was examined to determine (a) the correlation between 
the total score on each form of the developmental edition of the speaking 
test and the total score on the FSI and TOEFL, (b) the correlation between 
each item type and the FSI and TOEFL part and total scores, and (c) the 
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correlation beti^en each item and the FSI and TOEFL part and total acores* 
A special Item analysis program, SCALAR, laentlfied the 48 items from each 
form having the highest correlations with the total FSI score; also, a 
comparable analysis was performed using the total of TOEFL Section II 
(Structure and Written Expression) and Section III (Reading Comprehension 
and Vocabulary) in place of the total score, since TOEFL Section I (Lis- 
tening Comprehension) was expected to correlate highly with speaking 
proficiency. 



Results 



Based on the correlational analyses, four item formats were selected 
to form a 20--item prototype Test of Spoken English (TSE), which showed an 
internal consistency reliability of .91. The lt€an formats and scoring 
criteria selected were (a) reading aloud (pronirociation and fluency); (b) 
sentence completion; (c) telling a story based on a picture sequence 
(pronunciation, grammar, and fluency); and (d) nailtiple questions based on 
a single picture (grammar and fluency). The prototype TSE had equal 
correlations (.68) with FSI ratings and with TOEFL Structure and Written 
Expression and TOEFL Reading Comprehension and Vocabulary. While one 
might expect such a test to show higher correlations with the FSI inter- 
view than with TOEFL, the lower reliability (.74) of the FSI ratir^gs may 
have helped reduce the correlation between the TSE and the FSI. The 
addition of the prototype TSE to the TOEFL scores caused the multiple 
correlation with FSI to Increase to .73, thus Increasing the predictable 
FSI variance by about 8 percent. 



Conclusions 



The final prototype T,>E was presented to be considered for eventual 
operational use in the TOEFL program. Subsequently, a modified version of 
the TSE form developed in this study was pilots-tested under actual opera- 
tional conditions at representative TOEFL test centers. 



71 



-66- 



32. Gl 



20. Clark. J. L. D. , & Swlnton, S. S. (1980). The Test of Spoken English 
as a measure of cOBaaunlcatlve ability In English-medlua instruc- 
tional settings . (TOEFL Research Rep. No. 7; ETS Research Rep. 
No. 80-33) c Princeton, NJ: Educational Testing Service. (ERIC 
Document Reproduction Service No. ED 218 960) 



Purpose 



The two principal objectives of this study were (a) to determine the 
concurrent validity of the Test of Spoken English (TSE) in relation to the 
Foreign Service Institute (FSI) oral proficiency interview, and (b) to 
obtain criterion-related validity data on the relationship between TSE 
scores and actual communicative effectiveness in classroom settings. 



Method 



For the concurrent validation analysis, the TSE and the FSI interview 
were administered to 134 foreign teaching assistants at nine state univer- 
sities. Relevant background data, such as length of residence in English- 
speaking countries and years of English study prior to and subsequent to 
arrival in the U.S., were collected. Scores on the three-part TOEFL, 
obtained within the preceding 12 months [presumably via International or 
Special Center administrations], were available for 31 of these students. 
The TSE yielded scores in four areas: pronunciation, grammar, fluency, 
and overall comprehensibility . [The TSE is further defined in Summary No. 
19, Clark & Swlnton, 1979. J Correlations were computed among the TOEFL 
and TSE part and total scores and the FSI total score. 

In the criterion-related validation phase, FSI and TSE scores of 60 
foreign teaching assistants served as predictor variables in multiple 
regression analyses. Criterion variables were student ratings on a spe- 
cially designed questionnaire of the instructor's use of language in the 
classroom and in other instructional contexts, and a standardized instruc- 
tor/course evaluation instrument completed by the students. 



Results 



For the original group of 134 subjects the correlation between the 
TSE and FSI total scores was .79. For the subset of 31 cases for which 
TOEFL scores were available, the Intercorrelations were as shown in Table 
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Table 1 



TOEFL, TSE, and FSI Intercorrelations 
(N - 31) 



TOEFL TOEFL TOEFL TOEFL 
LC S&WE RC&V Total 



TSE TSE TSE 
Pron. Gram. Flu. 



TSE 
Ck)mp. 



TOEFL LC* 
TOEFL SoWE* 
TOEFL RC&V® 
TOEFL Total 



.77 
.67 
.92 
.68 
.76 
.65 
.69 
.71 



.64 
.91 
.42 
.54 
.52 
.46 
.57 



.85 
.38 
.56 
.43 
.36 
.62 



TSE Fron. 
TSE Gram. 
TSE Flu. 
TSE Comp. 
FSI Total 



.56 
.70 
.60 
.57 
.71 



.86 
.92 
.95 
.77 



.89 
.88 
.73 



.93 
.76 



.76 



*TOEFL LC - Listening Comprehension; TOEFL S&WE - Structure and 
Written Expression; TOEFL RC&V Reading Comprehension and Vocabulary. 

The TSE scores correlated more highly with the FSI rating t.ian with 
the TOEFL scores. The TSE gracamar score correlated more strongly with the 
TOEFL total (.70) than did the other TSE scores. The highest correlations 
Involving any individual TOEFL section were those among TOEFL Listening 
Comprehension, the four TSE scores, and the FSI total score. Thus, the 
TOEFL Listening Comprehension score showed a greater relationship to oral 
English skills than did the other two section scores. The TSE scores 
showed higher correlations with the FSI total than did any TOEFL score. 

The multiple regression analysis showed that both the TSE and the FSI 
scores served as good predictors of comiownlcative proficiency as deter- 
mined by student ratings on the special questionnaire; these test scores 
also predicted more general aspects of teaching performance. The TSE 
score correlated with proficiency in delivering classroom lectures (.60), 
understanding student questions (.52), answering questions (.53), and 
communication during office appointments (.54). The TSE score also 
correlated (.68) with the degree to which the instructor's pronounciatlon 
interfered vith student understanding. To a lesser extent the TSE score 
correlated with the instructor 'p "overall teaching effectiveness" (r^ - 
.29), and with other nonllnguistic variables, such as organization and 
planning, interpersonal relations, assignments and workload, and evalua- 
tion procedures. While FSI scores slightly exceeded TSE scores as predic- 
tors, both the TSE anC FSI scores showed appreciably higher correlations 
with communicative effectiveness than did the background variables. 
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Conclusions 



The high correlation between TSE and FSI scores indicates that the 
TSE may be considered a reasonable alternative to the FSI interview when 
it is not possible to carry out face-to-face testing. While the TSE 
grammar score is closely associated with TOEFL scores, the TSE pronuncia- 
tion and fluency scores appear to loeasure somewhat different aspects of 
language proficiency than those measured by TOEFL or by the TSE grammar 
score. 
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21. Cowell, W. R. (1981, April). Applicability of a simplified three- 
pararoter logistic model for equating tests . Paper presented at 
the meeting of the American Educational Research Association, 
Los Angeles. 



Purpose 

This study examined differences in TOEFL scaled scores produced by 
various methods used for equating TOEFL forms. The traditional linear 
equating method was compared with three different equating methods based 
on item response theory (IRT). For each model, equating was done based on 
both large samples and small samples to determine the importance of sample 
size in equating. Although data from the Secondary Level English Profi- 
ciency, or SLEP, test were also exmnlned, this sianmsry focuses on analysis 
of the TOEFL data only. 



Background 

Each TOEFL is equated to previous tests, to ensure that a given 
scaled score on one test is equivalent to the same scaled score on an- 
other. Until September 1978, TOEFL forms were equated by means of a 
linear equating model. Each test contained items from pretesting sections 
of previous tests, and these common items formed a babis for transforming 
scores on a given test to the scale used for previous tests. According to 
this model, a linear relationship was assumed to exist between the number 
correct and the scaled score. Since September 1978, equating methods 
based on the three-parameter model of IRT have been used as outlined by 
Lord.^ This model assumes that, for each item on a test, the probabil- 
ity that an examinee with a given ability level will answer the item 
correctly is mathematically related to three parameters: (a) a measure of 
item discrimination, (b) a measure of item difficulty, and (c) the prob- 
ability that a very low ability examinee will answer that item correctly. 
Statistics for the entire test are a composite of statistics for the items 
making up that test. 

A second possible equating model is the simplified IRT model, which 
is the same as the three-parameter model, except that parameters "a" and 
"c" above are assigned constant values across all items^ and only the 
item-difficulty parameter is assumed to vary. A third possible model is 
the Rasch one-parameter model, in which constant values are again assigned 
to parameters "a" and "c" (although not the same constants as in the 
simplified IRT model), and only the item-difficulty parameter is assumed 
to vary. 



^Lord, F, M, (1980), Applications of item response theory to practl-* 
cal testing problems . Hillsdale, NJ: Lawrence Erlbaum Associates. 
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Method 



The subjects were subsets of examinees who took TOEFL at an Inter*^ 
national or Special Center administration In September 1976, September 
1977, and August 1978. Numbers of subjects in the large saaple analysis 
ranged from 2,069 to 3,172 examinees per test; lis in the ^all sample 
analysis ranged from 292 to 317 examinees per test. 

Item calibration (i.e^, derivation of IRT parameters for each item) 
and test equating were done separately for each of the three test sec** 
tlons^ Each section of the 1977 test form and the 1978 test form were 
equated to the 1976 form, for Iwth the large sample and the small sample, 
using each of four different methods: (a) the linear iMtthod, (b) the 
three-parameter IRT method, (c) the simplified IRT method, and (d) the 
Rasch one'-parameter IRT method. The resulting scaled scores were then 
examined, for each item in the test, to determine the degree of cor re** 
spondence in results obtained according to the different equating methods. 



Results and Conclusions 



The differences in results obtained according to the linear equating 
and IRT equating models were relatively small over most of the score range 
for all three of the IRT equating models and for both the small and large 
sample sizes. On the average, the scaled score for a given raw score 
derived by these various methods differed by less than one point on the 
TOEFL scale. Thus, all of these equating mettods evidently wrked reason-- 
ably well and produced comparable results. 

The simplified IRT model and the Rasch one^parameter IRT model 
produced results more closely approximating those produced by the linear 
model than did the three^-parameter model. This is due to the fact that 
the three-parameter model resulted in a somewhat more curvilinear function 
relating raw and scaled scores; raw scores below a certain amount yielded 
slightly higher scaled scores, and raw scores above a certain amount 
yielded slightly lower scaled scores, than would be predicted by a linear 
relationship. As a result, the standard deviations of scaled scores based 
on the three^^parameter model were about 6 percent smaller than those based 
on the linear model. 

Comparison between pairs of IRT models yielded differences about 
one-*half the magnitude of the differences between the linear model and 
each of the IRT nH>dels. Also, differences due to variation in model were 
about twice as large as differences due to variation in sample size. 
Estimation of costs required to equate tests according to each model 
showed that a more substantial reduction in cost wo\ild be achieved by a 
decrease in sample size than by a change from the threes-parameter to the 
one-parameter model. 
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In conclusion, if the large-sample, three-paraaeter IRT approach now 
in use were to be replaced in order to simplify the task of test equating, 
the data suggest that it would be better to focus on reducing the sample 
sl^e than on reducing the ntsaber of parameters in the model. 
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22* Cowell, W. (1982). Itea-response-theory pre-equatlng In the TOEFL 
testing program. In P« W« Holland & D« B« Rubin (EdsO^ Test 
equating (pp* 149-161) • New York: Academic Press* 



This chapter describes the historical background of TOEFL test 
equating, the initial conversion to item-response-theory (XRT) equating, 
and the principal advantages of the use of IRT in determining scores on 
different forms of TOEFL. 



A new form of TOEFL is given each month at test centers in 135 
countries. To ensure equivalent scores for persons of equal ability, 
regard: ess of the level of difficulty of the particular form of the test 
they were administered or t*he level of English proficiency of those with 
whom they took the test, each new form of TOEFL is equated to previous 
forms. A score is reported for each section on a scale ranging from 2r to 
80. The total score equals ten-thirds times the sum of the scaled scores 
for the three sections. Prior to September 1978, equating was done by 
including in each new form a set of items from a previously equated form. 
Because the equating items had been used in a previous test, there was a 
danger that they might appear in commercial coaching materials or that 
they might be seen by test repeaters. In order to Improve test security 
and thus the accuracy of the equating process, it was decided to switch to 
IRT equating methods. 



In July 1977 a feasibility study was initiated. The first phase 
involved the IRT equating of all TOEFL forms administered between Septem- 
ber 1976 and May 1977, plus three experimental forms. The items in these 
forms and the items in several pretests were all calibrated to a common 
scale using a three-parameter model of item performance. In the second 
phase, new forms administered between September 1977 and August 1978 were 
equated by both conventional (linear) and IRT methodology. 



Score conversion tables show that IRT equating produced scores that 
were nearly identical to those obtained by linear equating. The largest 
differences were typically about two points per section, except at the 
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chance-8Core range* where differences as large as four paints may have 
appeared. These differences cannot be considered error for either method, 
since there is no way to judge the superiority of one method over another* 
Differences were particularly small in the mid-scale range* 



Conclusions 



The fact that scores were nearly identical and that differences were 
smallest in the mid-^scale range, which is usually where the critical 
decision-making point is located, means that it«a-response-theory can be 
employed operationally for TOEFL equating* The theory offers several 
advantages that make it attractive for this purpose* IRT equating infor- 
mation can be obtained from different pretests that can be given to 
relatively small samples (1,000 to 2,000) located at the most secure test 
centers* For tests consisting entirely of precalibrated items, equating 
can be done before the operational form of the test is administered, 
thereby reducing the number of operations to be performed during the 
period between test administration and score reporting* Finally, unlike 
linear equating, IRT equating produces measurement er^br that is not 
cumulative over forms. 
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23. Darnell^ D. K. (1970) « Clozentropy: A procediure for testing English 
language proficiency of foreigti stiidents* Speech Monographs t 
37 » 36-*46. Also printed as **The development of an English 
language proficiency ttst of foreign students, using a closen- 
tropy procedure.** Final Report, U.S. Office of Education 
Project No. 7-H-OlO, 1968* (ERIC Document Reproduction Service 
No. ED 024 039) 



Purpose 



A procedure for obtaining scores on a cloze test was examined. 
Entitled '^clozentropy,'* the procedure. In effect, measures the comparabil- 
ity of a foreign student^s English with that of native speakers of 
English. For graduate and undergraduate students separately, clozentropy 
scores were compared with scores on TOEFL, which served as a criterion 
measure of English proficiency. 



Method 



The subjects were 48 foreign students at the University of Colorado — 
21 undergraduates, and 27 graduate students. Approximately half the 
subjects In each group were engineering majors and half were nonenglneer- 
ing majors. Also, 200 native speakers of English at the same university 
were given the clozentropy test (half undergraduate and half graduate and, 
within each group, half engineering majors and half nonenglneerlng ma- 
jors). The samples were chosen at random, with supplementation from 
volimteer subjects. 

The foreign-student sample was administered the five-part TOEFL in a 
special testing session at the University of Colorado. Within a month 
after the TOEFL administration, all subjects were administered the clozen- 
tropy test (all but four of the foreign students took the clozentropy test 
within a week after TOEFL, in one of two special group administrations). 

The clozentropy procedure can be described as follows. As in the 
typical cloze test, subjects (in both the foreign student sample and the 
native speaker sample) read a passage from which words were systematically 
replaced with blank spaces, and they sought to identify the words that had 
been deleted. For the native English speakers, the frequencies of differ- 
ent responses to each blank were computed and, for each blank, an '*H*' 
value ( "entropy*") was computed, which essentially varied with the number 
of alternative responses to that blank. Then, for every student, an "I" 
value was computed for each response, which reflected the rarity of that 
response In the native speaker sample (i.e., a high 1 value indicated that 
the response was rare; a low value showed that the response was common). 
Finally, for each subject, a deviation score, H minus I, was computed for 
each blank. The sum of a subject's deviation scores for all blanks 
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cottstituted that subject's clozentropy score. In effect, then, the higher 
8 subject's clozentropy score, the more the subject's responses conformed 
id.th those of the native English speakers. 

Four test passages were used in the study, each 500 words in length: 
an easy and a difficult passage from engineering texts, and an easy and a 
difficult passage from liberal arts texts. For each subject, a clozentropy 
score was obtained for each of these four passages as well as a total 
clozentropy score. 

In addition to the clozentropy scores and scores on TOEFL (five part 
scores and total), each subject's grade-point average (GPA) was obtained. 

Results 

Reliability coefficients for the foreign student sample, computed by 
a method based on analysis of variance, were .86 for both the clozentropy 
test and TOEFL. Intercorrelations among scores for the four different 
clozentropy passages ranged from .53 to .65, providing additional evidence 
for the reliability of the clozentropy test. 

Intercorrelations were computed among clozentropy scores, TOEFL 
scores, and GPA for the foreign students. A relatively high correlation 
of .84 was obtained between total clozentropy score and total TOEFL score 
(.88 for graduate students alone, .78 for imdergraduates alone). Correla- 
tions between total clozentropy and the TOEFL subtests were all signifi- 
cant (all above .61). Interestingly, the Listening Comprehension subtest 
of TOEFL correlated about as highly with the clozentropy score (.74) as it 
did with the total TOEFL score (.75). GPA failed to correlate signifi- 
cantly with the clozentropy test or with any of the TOEFL subtests or 
total TOEFL score. 

Analyses of variance assessing main effects and interactions involv- 
ing seven different factors on the clozentropy scores were also computed. 
Because this portion of the study does not bear on TOEFL, results of these 
analyses will not be enumerated here, except to note that many variables, 
including the subject's level of English proficiency, the subject's major 
field, the context of the passage, and the difficulty of the passage, had 
a main effect or interaction effect on the clozentropy scores. It appears 
to be lmporta.it, therefore, to consider the specialized needs of individu- 
al students in testing language proficiency. 



Conclusions 

The relatively high reliability coefficients for both TOEFL and the 
clozentropy score provide support for the reliability of the clozentropy 
score. The clozentropy battery took only half as long to administer 
88 TOEFL, and further increases in reliability might be achieved by 
increasing its length. 
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24. Dizney, H, (1965). Concurrent validity of the Test of English as a 
Foreign Language for a group of foreign students at an American 
university. Educational and Psychological Measurement , 25 , 
1129-1131. 



Purpose 

This study was an early attempt to Investigate the correlational 
validity of TOEFL. At the time of the study, published Information about 
TOEFL validity Included only correlations with instructors' ratings of 
language proficiency. 



Method 

The sample of students included 20 foreign students at Kent State 
University (19 males and one female) representing 15 countries. At the 
time of testing, 13 subjects had been in the United States for less than 
one month. Their ages ranged from 19 to 38, with a median of 23.5 years. 

In September 1964 the subjects took the five-part TOEFL, the three- 
part Michigan Test of English Language Proficiency (MTELF) , and the 
English subtest of the American College Test (ACT). The MTELP is a test 
of English proficiency consisting of three sections: Grammar, Vocabulary, 
and Reading Comprehension. [See Summary No. 1, Abadzi, 1976, for a 
further description of the test.] The ACT English subtest is part of a 
battery of tests for American students applying to U.S. colleges and 
universities and taps knowledge of English usage. In recent research it 
has shown a reliability of .84 and a correlation of .57 with grades in 
freshman English courses. In addition, the subjects were rank ordered 
with respect to English language proficiency by the foreign student 
adviser; this was done six weeks after the start of the fall quarter. 



Results 

The means and standard deviations for the tests were TOEFL: 538 and 
106; MTELP (raw score): 79 and 18; ACT English (raw score): 31 and 14, 
Correlations among these tests were MTELP-TOEFL: .97; MTELP-ACT English: 
.65; TOEFL- ACT-Engllsh: .74. Rank-order correlations between English 
language proficiency ratings and ranks on each standardized test were 
as follows: MTELP; .78, TOEFL: ,75, and ACT English: .62. 



Conclusions 

For this sample TOEFL and the MTELP were highly correlated (£ .97), 
Thus, use of both tests to assess English proficiency may be reidimdant, 
TOEFL accounted for almost 13 percent more variance on the ACT English 
subtest than did the MTELP, The difference between TOEFL and MTELP 
correlations with the foreign student adviser's rankings of English 
language proficiency was negligible. 
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25. Dohsrty, C, & Ilyln, D. (1981). Technical manual for ELSA: English 
language skills assessment in a reading context . Rowley, MA: 
Newbury House • 

Purpose 

This manual describes the development ot a set of tests entitled 
English Language Skills Assessment in a Reading Context (ELSA) and pro- 
vides statistics from several samples showing the reliabilities of the 
tests ana correlations with other measures of English language profi- 
ciency, including TOEFL. The ELSA tests were designed to meet a placement 
need for the more rhan -^0,000 adults who enroll annually in English as a 
second language (ESL) classes in adult schools in San Francisco. 



Method 

Data are reported for several different testing samples, drawn from 
adult ESL programs in San Francisco and from intensive English programs at 
the University of San Francisco and San Francisco State University. 

The ELSA tests use a mul tiple-^choice cloze format* Five diflerent 
ZLSA tests were constructed; two beginuiag ti:Sts, two intermediate tests, 
and oae advanced tes For the beginning and intermediate levels, one 
test used a passage w ha conversation ^'ormat (tests BC and IC), and 
another test used a passage w^th a narrative format (BN and IN). The 
advanced test used only a narrative format (AN). 

Eac*^ test was constructed by writing passages in which every seventh 
word elicited a specific asptct of structure or semantics; deletion of 
these words produced 3? bla- rs. Testing with several hundred students 
led to selection of the best 25 items (such that no more than lA words 
appeared between blanks) as well as identification of frequent errors, 
which then served as diatractors in the final tests. The final version 
of each test consisted of 25 items aud required about 30 minutes to 
administer. 

The ELSA tes' e compared with several other tests of English 

proficiency. One s test waB IDEVL [presumably the three-part version] 
administered through tne International/ 3pecirl Center programs. Another 
was thr Comprehensive English Language Test for Speakers of English as a 
Second ^nguage (CELT). Two subtests of the CELT were used. The Structure 
subtest consists of 75 multiple-choice items in a reading format. The 
Listening subtest contains 50 items; for each item the subject listens to 
a script and responds to one of several printed alternatives. [For a 
further description of the CELT, see Summary No. 41, Hosley & Meredith, 
1976,] The Listening Comprehension Picture Test (LCPT) is a multiple- 
choice listening test in which the subject listens to orally presented 
information, then hears a question and identifies the one picture of five 
that depicts the correct answer. The Michigan Test of x^nglish Language 
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Proficiency (MTELP) (Forms A, B and C) tests grammar, vocabulary, and 
reading comprehension. (For a fuller description see Summary No. 1, 
Abadxi, 1976.1 In the Structure Tests for English Language (STEL) (inter- 
mediate forms), students read and respond to 50 multiple-choice items. 

Data were collected on different occasions from approximately 10 
different samples of subjects between the fall of 1978 and the summer of 
1980, for a tctai of several hundred subjects. 



Results 



The first results of interest concern reliabilities of the ELSA tests 
(KR-20 formula was used for beginning tests and KR-21 for other tests). 
The two beginning tests (BC and BN), for which data from over 600 subjects 
were available, yielded reliabilities in the low 80s. The combination of 
the two beginning tests yielded a reliability of .92. Of the two inter- 
mediate tests, test IN yielded reliabilities of .79 and .80 in two sepa- 
rate samples (Ns - 73 and 158, respectively). Test IC yielded reliabili- 
ties of .76, .80, and .82 in three separate samples (IJs - 73, 212, and 
115, respectively). Combination of the two Intermediate tests yielded a 
reliability of .88 in each of two samples (^s - 73 and 105, respectively). 
The advanced test showed reliabilities of .80 and .83, respectively, when 
given to (a) a combination of adult center and intensive English classes 
(!J - 250), and (b) to a later sample of intensive English classes (£ - 
115). Reliabilities computed separately for each level of English lan- 
guage study tended tc be somewhat lower, due to restriction of range. 
Combinations of tests generally yielded higher reliabilities than did 
single tests. 

Correlations between the LCPT (25-ltem version) and the two ELSA 
beginning tests were significant and moaerately high, ranging from .58 
to .61 (N « over 400 students). The correlation between the two beginning 
ELSA tesTs (BC and BN) was .82 (N - 664). 

Correlations between the ELSA intermediate tests and scores on the 
LCPT (33-item version) were significant and ranged from .53 to .58j 
correlations between the ELSA Intermediate tests and the STEL were .79 to 
.82 (N - 73 for both sets of correlations). The ELSA IC test correlated 
.73 with the MTELP and .83 with TOEFL (N • 113). [The relations between 
test IN and the other tests are not reported.] The correlation between 
the two intermediate ELSA tests (IC and IN) was .84. l£ not indicated.] 

The advanced ELSA test (AN) correlated .80 with CELl -Structure (N « 
77), .56 with CELT-Llstening (N - 75), and .82 with the MTELP (N - 113). 
Correlations between the ELSA advanced test and TOEFL in two separate 
samples were .73 and .84 (^s - 77 and 29, respectively). 
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Conclusions 



Cor rela{;lons between the ELSA tests and other tests of English 
proficiency were relatively high> attesting to the validity of the ELSA 
tests. The relatively high correlations between forms of the l^ginning 
test suggest that these tests can be substituted for each other. The same 
is true for the two forms of the intermediate test. Also, the correla- 
tions between the intermediate tests and the STEL (.79 and .82) were high 
enough to suggest that the Intermediate ELSA tests, which are shorter, 
could be used in place of the STEL if testin? time were limited. The 
advanced test correlated highly enough with CELT'-^St rue ture (.80) to 
substitute for this measure; the same was not true, however, of CELT- 
Listening. 

The correlations between the advanced test and TOEFL (.73 and .84 for 
two samples) are suggestive. Conclusions about the magnitude ot relation- 
ship between these tests, however, must await research with larger 
samples. 
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26. Dunbar, S. B. (1982). Construct validity and the Internal structure 
of a foreign language test for several native language groups . 
Paper presented at the annual nseetlng of the American Education- 
al Research Association, New York. 

Purpose 



Confirmatory factor analysis was used to examine the factor structure 
of TOEFL for each of seven language groups. The objectives were to 
determine (a) whether performance on TOEFL reflects the content specifica- 
tions of the test, as shown in the test's factor structure, and (b) 
whether there are differences among language groups in this respect. 



Method 



The subjects were examinees taking the September 1976 administration 
of the three-part TOEFL. There were approximately 600 to 1,000 subjects 
froffl each of the following language groups: African, Arabic, Chinese 
(non-Taiwanese), Farsi, Germanic, Japanese, and Spanish. The African 
languages consisted of Eflk, Fanti, Ibo, and Yoruba; the Germanic consist— ' 
ed of Danish, Dutch, German, and Swedish. (The sample was the same as that 
used in the study by Swinton and Powers, 1980.) A reference sample of 
close to 1,000 subjects drawn from all language groups in the test admin- 
istration sample was also included in the study. 

For the present analyses the test was scored in the following manner: 
for Listening Comprehension, seven different composite scores were 
derived, two or three composite scores for each of the three different 
subtypes of items used in this section, according to TOEFL item specifica- 
tions; for Structure and Written Expression, four different composite 
scores were computed, one for each of the four item subtypes; and for 
Reading Comprehension and Vocabulary, seven composite scores, three or 
four ff^r eaca of the two item subtypes. 



Results and Conclusions 

The language groups differed from each other in total TOEFL scores, 
with the Germanic group differing most dramatically from the rest, shcvlng 
the highest mean score and lowest variability in scores. 

Confirmatory factor analyses were conducted. Estimates of factor 
loadings were obtained for three hypothetical models: (a) a null model, 
assuming complete independence of subscores; (b) a model with one general 
factor; and (c) a model with four factors — a general factor r.actor 
I) plus one factor corresponding to each of the three TOEFL subtests 
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(Factors II, III, and IV) ♦ Goodness of fit analyses showed that the model 
with one general factor accounted for the subjects' performance much more 
accurately than did the null model, and that the four-factor model provid-* 
ed an even better fit to the data. (Note that the issue regarc'ing the 
total number of factors needed to account for the data is not addressed 
with confirmatory factor analysis.) 

In the analysis Involving four factors, intercorrelations among the 
four factors were examined for each of the language groups. For the 
African group, intercorrelations among Factors II, III, and IV were 
relatively high, particularly the correlation of Factor IV with Factor II 
(•83) and Factor III (.92)^ For the Arabic, Chinese, and Germanic groups. 
Factor III showed moderate to high correlations (.48 to .82) with Factois 
II and IV. These data suggest that, for different language groups, 
different numbers of factors may be required to account for the data. 

The issue of language group differences in factor structure was 
addressed in other analyses as well. Targeted rotations of factor pattern 
matrices were derived to allow comparison of the factor structure for each 
language group with that for the reference sample. The resulting "ccn-- 
gruence coefficients'* in comparisons with the reference sample were 
uniformly high for Factor I, the general factor (coefficients ranged from 
.991 to ,998, wich 1.000 as the maximum possible coefficient), and for 
Factor II, the factor corresponding to the Listening Comprehension section 
(coefficients ranging from .969 to .998). For Factor III (which corre- 
sponded to the Structure and Written Expression section), congruence 
coefficients were high for three language groups, Arabic, Chinese and 
Germanic (.959 to .995), but not nearly so high for the Japanese (.615), 
African (.859), Spanish (.881), and Farsl (.906) groups. This result 
calls into question the assumption of equivalence of Factor III across 
language groups. 

For Factor IV (which corresponded to the Reading Comprehension and 
Vocabulary section), all congruence coefficients were above •985 except 
those for the Spanish group (.870) and the African group (.932). Thus, 
for Factor IV, congruence with the reference sample se^ed generally high, 
although so le language differences involving this factor were evident. 

Further analyses helped to determine the role and interpretation of 
the various factors. The general factor dominated the factor structice 
for all language groups. After the general factor was partialed out, the 
Listening Comprehension items supported a relatively large factor (Factor 
II); the seven different composite scores (types of content) in the 
Listening Comprehension section appeared to be about equal to each other 
in their degree of loading on this factor. Factor III was a much less 
prominent factor, as leadings of the composite scores on this factor were 
as small as one'**eighth the size of their loading on the general factor; as 
with Listening Comprehension^ the different cc posite scores in the 
*">tructure and Written Expression section appeared to be about equal in 
their degree of loading on Factor III. For Factor IV, the largest load- 
ings were for vocabulary item types, as the loadings of reading comprehen- 
sion item types were very small (the reading itea types, instead, loaded 



heavily on the general factor). Thus, Factor IV appeared to be basically 
a vocabxilary factor. 

In general, the data show the dominance of the general factor and, at 
the same time, show the importance of factors associated with different 
sections of the test. The data also provide aoderate support for the 
notion that the factor structure of TOEFL is similar across language 
groups. This similarity appears to be due primarily to the general factor 
and, to a lesser extent. Factors II and IV. Although there were some 
language group differences involving Factor III, this factor itself was 
quite small in relation to the general factor, thus tending to lessen the 
impact of any group differences. Despite these observations, evidence of 
fKt least some differences among language groups in nmber and nattire of 
factors suggests the possibility, witlch deserves further investigation, 
that TOEFL subtest scores may have differential validity for different 
language groups. 



SS 



27. Educational Testing Service. (1973) • Manual for TOEFL score recipi- 
ents, Princeton, NJ: Author. 



Purpose 



This document is an earlier version of the T OEFL Test and Score 
M anual (Educational Testing Service, 1981) summarized elsewhere in this 
collection. This 1973 edition is the most recent manual that deals with 
the five-part TOEFL and is summarized here primarily to present statisti- 
cal data for that version of the test. In most other respects the two 
editions of the manual are similar, so the narrative in this edition is 
summarized only where it differs from that presented in the 1981 version. 



Discussion 



In 1973 TOEFL could be taken at International, Institutional, and 
••walk-in" administrations. In the International Testing Program, TOEFL 
was administered on four test dates per year. The Institutional Testing 
Program was offered only within the United States and Canada. "Walk^in" 
test centers were established at six locations in the United States at 
which TOEFL was administered, usually on a weekly basis, as a service to 
institutions with an immediate need for TOEFL scores of applicants 
currently in the United States. 

Statistical characteristics of the test were examined for all forms 
of TOEFL administered from October 1966 through June 1971. Mean KR-20 
reliabilities for the five sections and total were Listening Comprehen-* 
sion, .90; English Structure, .86; Vocabulairy, .89; Reading Comprehension, 
.84; Writing Ability, ,86; and total score, .97. ^fean standard errors of 
measurement for he five sections and total, respectively, were 3.0, 2.9, 
2.8, 3.3, 2.9, and 14.77. This last figure indicates that any two exam- 
inees* TOEFL scores should not be Interpreted to represent different 
levels of proficiency unless the difference between their scores was 
greater than about 30 points I see Summary No« 28, Educational Testing 
Service f 1981, for explanation]. 

Intercorrelations among subtests, averaged over the forms mentioned 
above, were as shown in Table 1. 
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Table I 



Intercorrelations among TOEFL Subtests 



TOEFL Subtest 



LC 



ES 



V 



RC 



Listening Comprehension (LC) 
English Structure (ES) 
Vocabulary (V) 
Reading Comprehension (RC) 
Writing Ability (WA) 



.64 
.56 
.65 
.60 



.72 
.67 
.78 



.69 
.74 



.72 



Tables are also presented in the manual showing the percentile ranks 
corresponding to various TOEFL scores » broken dovm by intended major 
field. The data are ba^ed on the 215, 48f^ examinees seeking admission to 
U.S. colleges and universities who took ..OEFL from October 1966 through 
June 1971. The data are tabulated separately for (a) graduate candidates, 
(b) undergraduate t ndidates, and (c) all candidate for total TOEFL 
score and for each section. Also, TOEFL score means are presented for 
each native language and each native country represented. 

The use of TOEFL scores is discussed, and a table is presented 
showing data from a 1969 survey of institutions regarding the role of 
TOEFL in their admissions practices. Presented separately for institu- 
tions with and without special ESL courses, and separately for undergradu- 
ate and graduate applicants, are the numbers of institutions for whom the 
lowest acceptable score ranges for admission of candidates were (a) below 
400, (b) between 400 and 449, and so forth. Also, a composite of the 
admissions policies of two U.S. universities with large numbers of foreign 
students is provided, which shows the kinds of decisions associated with 
each of several ranges of total TOEFL scores. Decision possibilities 
include admission with or without restrictions depending on the students' 
patterns of TOEFL section scores, their educational levels, and, for 
graduate students, their intended major fields. 

In other respects, the information presented in the manual is gerer- 
ally similar to that presented in the 19|B1 version, including a b' lef 
overview cf TOEFL's early history, a description of the test, illustration 
of the score report, and a discussion of guidelines for proper use of 
TOEFL scores. 
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28. Educational Testing Service. (1981). TOEFL test and score manual . 
Princeton, NJ: Author. 



Purpose 



This manual presents a description of TOEFL, explains the operation 
of the TOEFL program, and provides information and statistics relevant to 
interpretation of TOEFL scores. 



Discussion 



An overview of the TOEFL program is offered first, with a brief 
account of the history of TOEFL, beginning with its initial development 
in 1963. The three subtests (sections) of the current TOEFL are de- 
scribed, and the process of constructing test questions is briefly dis- 
cussed. The procedures for reporting test results are outlined with an 
illustration of the TOEFL score report and an explanation of the methods 
for deriving section scores and combining section scores into total 
scores. [See Introduction for a description of the three-part TOEFL and 
for reference to summaries detailing TOEFL 's history.] 

The four major TOEFL testing programs are described. These include 
the International and Special Center testing programs, each of which 
provides test administrations (typically every other month) for persons 
seeking admission to institutions in the United States or Canada or 
seeking professional certification. Also Included are the Institutional 
Testing Program and the Overseas Institutional Testing Program, which 
provide previously used TOEFL forms to institutions for testing their own 
students, principally for placement in English courses or determining the 
need for additional English study. 

In a discussion of the use of TOEFL scores, the following guidelines 
are presented: (a) base the evaluation of an applicant on all available 
relevant information; (b) do not use rigid "cut-off" scores in assessing 
an applicant's TOEFL performance; (c) consider scores on the three test 
sections as well as the tofal scores; (d) consider the kinds and levels of 
English proficiency required in different fields and levels of study and 
the admitting institution's capacity to provide English language training; 
(e) consider TOEFL scores in interpreting an applicant's perforaance on 
other standardized tests; (f) do not use TOEFL scores for predicting 
academic performance; and (e) gather information about the validity of 
TOEFL score requirements at the admitting institution. 

Resiats of a 1980 survey of institutions regarding their use of TOEFL 
scores in admissions decisions are provided. Data from 386 respondents 
(principally four-year undergraduate institutions and graduate and 
professional schools) show the kinds of TOEFL score ranges considered 
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acceptable for making various decisions, including (a) admit with no 
restrictions, (b) admit on part-time basis with supplementary English 
instruction, and (c) refer to full-time English program. 

Data for a reference group (N - 473.944) of examinees taking TOEFL 
between September 1978 and August 1980 are presented. Mean scores were 
Listening Comprehension, 51.5 ( SD - 6.9); Structure and Written Expres- 
sion, 48.6 (SD - 7.8); Reading Comprehension and Vocabulary, 49. ^MJ - 
7.2); and toTil score, 499 (SD - 67). Percentiles corresponding to 
various scores are provided, and breakdowns are given by planned level of 
study (or licensure) and by examinees' native language and native country 
or region. 

Statistical characteristics of the test are provided for the seven 
forms of the test administered between February 1980 and August 1980. IN 
not indicated.) The data show the difficulty levels of these forms to be 
appropriate, as the average raw score per section ranged from 54 to 69 
percent correct (62.5 percent provides the best measurement). (Note that 
differences among forms are corrected in computing the scaled scores 
reported to students and institutions.) Speed was apparently not a major 
factor, as the data met the conditions used by Educational Testing Service 
to determine adequacy of time allowed, namely, (a) 80 percent of examinees 
should finish nearly every question in each section, and (b) three-fourths 
of the questions in a section should be completed by nearly all examinees. 

Reliabilities (KR-20) for the seven forms ranged from .86 to .89 for 
Listening Comprehension, .81 to .87 for Structure and Written Expression, 
.86 to .90 for Reading Comprehension and Vocabulary, and .93 to .95 for 
total score. Standard errors of measurement, averaged across forms, were 
2.2 for Listening Comprehension, 2.9 for Structure and Written Expression, 
2.5 for Reading Comprehension and Vocabulary, and 14.6 for total score. 
The last figure shows that an examinee's "true score" (i.e., hypothetical 
score that would be achieved if there were no errors of measurement) is 
within 14.6 points of the observed score for two-thirds of the examinees, 
and double this figure, or about 29 points, for 95 percent of the examin- 
ees. Thus, two examinees' scores cannot be regarded as representing 
different levels of proficiency unless therft is a difference of at least 
29 points between them. 

Averaged over the seven forms, Listening Comprehension correlated .70 
with Structure and Written Expression and .68 with Reading Comprehension 
and Vocabulary; the latter two subtests correlated .77 with each other. 
Thus, there appears to be a reasonably strong relationship among skills 
tapped by the different sections, but the section scores still provide 
some unique information. 

The question of validity is considered, with reference to several 
studies summarized in this collection as well as some unpublished data. 
In general, the studies examined suggest tliat (a) TOEFL correlates rela- 
tively highly with other standardized measures of English proficiency; (b) 
relevant sections of TOEFL correlate reasonably highly with direct mea- 
sures of writing ability; (c) TOEFL discriminates effectively among 
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foreign students but not among native speakers of English; and (d) the 
discrimination power of standardized aptitude tests Is greatest for 
students who do not have low TOEFL scores • 

Brief additional sections of the manual discuss procedures at TOEFL 
test centers, describe other tests developed by the TOEFL program, and 
list various TOEFL publications, order forms, and research reports. 
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29. Flahive, D, E. (1980). Separating the £ factor from reading compre- 
hension. In J. W. Oiler. Jr., & K. Perkins (Eds.), Research in 
language testing . Rowley, MA; Newbury House. 



Purpose 



Two questions of interest in this study were: What is the role of 
intelligence in language comprehension? and, To what extent is reading 
comprehension associated with intelligence? Information relevant to these 
questions was obtained by exaninins the interrelationships among a nonver- 
bal IQ test, TOEFL, and three reading tests. 



Method 

The subjects were 20 students, representing seven different native 
languages, in a semi-? 'tensive English class. The subjects' TOEFL scores 
averaged 509 and range^ trom 437 to 568. 

Five tests were administered within a one-week period. The Raven's 
Progressive Matrices Test (Standard Form), a measure of nonverbal intelli- 
gence, requires the subject to examine a pattern that has a portion 
missing and to choose, from asaong eif^ht alternatives, the one that best 
completes the pattern. The five-part TOEFL was administered; only the 
total score was used in this study. The first of three reading tests was 
the McGraw-Hill Basic Skills System Reading Test. It consists 10 
passages, five of approximately 250 words each, and five of approximately 
75 words each, followed by 30 multiple-choice questions. This test 
measures the subject's ability to make inferences, to identify main ideas 
and supporting points, and to discover organizational patterns. The 
second reading test, the Perkins-Yorlo test, contains 50 items; In each 
item the subject reads a sentence and then chooses, from among four 
alternatives, the sentence that best paraphrases the target sentence. The 
third reading task was a cloze test that contains a passage of approxi- 
mately 400 words from an Introductory economics text. Every seventh 
word has been deleted, to yield a total of 50 blanks, and, for each blank, 
the subject is required to determine the missing word. The appropriate- 
word scoring method was used; i.e., suitable substitutes for each deleted 
word were accepted as correct. 



Results and Conclusions 



The mean score on the Raven's test was just above the 75th percentile 
of the normlng group for this test, suggesting that the subjects were 
above average in nonverbal intelligence. Performance on tVie McGraw-Hill 
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test ranged from the 9th to the 85th percentile compared to a reference 
group of college freshmen and sophomores, Indicating a wide range of 
proficiency; consistent with this is the wide range of TOEFL scores 
reported above. Perkin-Yorio scores were only slightly lower than those 
of a reference group of college freshmen. 

Table 1 presents the intercorrelations among measures. 

Table 1 

Intercorrelations among Tests Used in the Study 



Raven's McGraw-Hill Perkins-Yorio 





Test 


TOEFL 


Test 


Test 


Raven's Test 










TOEFL 


.61 








McGraw-Hill Test 


.84 


.59 






Perkins-Yorio Test 


.68 


.84 


.67 




Cloze Test 


.61 


.75 


.65 


.62 



The relatively high correlations among most of these tests are not 
surprising, since most of the tests measure reading and/or language 
proficiency. However, the high correlation between the Raven's and 
McGraw-Hill tests is unexpected, since the former Js a measure of nonver- 
bal intelligence and would not be expected to be a strong predictor of 
reading performance. Single- and nmltiple-regression analyses showed that 
adding TOEFL to the Raven's test significantly improved prediction of 
scores on the Perkins-Yorio and cloze tests but not scores on the McGraw^ ' 
Hill test; this result again reflects the strong relationship between the 
Raven's test and the McGraw-Hill multiple-choice reading test. 

In general, the high correlations among tests observed here suggests 
that intelligence is a major component of performance on reading subtests 
In widely used measures of nonnative language skills. 
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30. Gales, S. J. (1976, May). Sentence-combining; A technique for 
assessing proficiency In a second language . Paper presented at 
the Conference on Perspectives on Language, University of 
Louisville, Louisville, KY. (ERIC Document Reproduction Service 
No. ED 130 512} 



Purpose 



Indirect measures such as those Included In TOErL measure the 
ability to recognize but not produce syntactically correct sentences, and 
written compositions directly measure a ctaabinatlon of many aspects of 
writing. The present study assessed the effectiveness of a rewriting task 
as a direct measure of a specific component of stirfents' writing ability: 
syntactic control. Performance on the rewriting task was examined in 
relation to performance on selected TOEFL subtests for learners of English 
as a second language. 



Method 



The study included a total of 41 subjects — 16 native speakers of 
English in graduate school at Indiana University, five highly proficient 
nonnatlve speakers in graduate school, and, in the principal sample, 20 
nonnatlve speakers of English in intensive English language classes. 

The subjec s were given the rewriting task developed by 0*Donnell and 
described by Hunt (1970).* The subJectP were given a passage consisting 
of many very short sentences, and they were asked to rewrite the passage 
by combining sentences, changing word order, and eliminating redimdancy. 

Three scoies were derived from this task, reflecting syntactic 
maturity of the subjects' rewritten passages: (a) words per clause, (b) 
clAuses per T-unit, and (c) words per T-unlt (which combines the first two 
scores). The T-unit is defined as "one main clause plus any subordinate 
clause or nonclausal structure that is attached to or embedded in it" 
(Hunt, 1970, p. 4). The subjects* scores on the five-part TOEFL were also 
available [prtsimiably from International admlnl8trati'>n8] . 



Results and Conclusions 



The native speakers in this study performed similarly to the tUlled 
adults in Hunt's study, and the five proficient nonnatlve speakers here 



^Hunt, K. W. (1970). Syntactic maturity in school children and 
adults. Monographs of the Society for Research in Child Developn^nt , 35.» 
(Serial No. 134). 



96 



perfornecl at an ev«.4 higher level* The target sample of 20 aonnatlve 
speakers t ho«)ever, showed an average level of syntactic complexity coups- 
rable to that of the seventh graders In Hunt's study (although a iflde 
range of scores was observed here). The average nuaber of words per 
clause for these aonnatlve speakers vas 6.25 (range: 4.59 to 8.57); 
clauses per T-unlt, 1.27 (range: 1.00 to 1.67); ard words per T~unlt, 7.96 
(range: 4.39 to 12.2^). 

Spearman rank correlations for the 20 nonnatlve speakers were cooput- 
ed among the three revnrlting scores, TpEFL total, and tvo TOEFL subtests, 
English Structure and Writing Abllll^y. Low correlations were observed 
between TOEFL English Structure and rcswrltlng scores (ranging fron .Ol to 
.32). This result suggests that the ability to recognise correct graa- 
laatlcal 8tr\:£ture Is not always accwapanied by proficiency In writing, and 
that active and passive skills In grammar are not necessarily related. 
Correlations involving TOEFL Writing Ability were .49 to .57, and correla^* 
tlons Involving TOEFL total were .23 'to .55. 

Definitive Judgments of the rewriting task's effectiveness cannot 
be made from this study, and later research should (a) look into refine* 
m^^nts in the scoring method, (b) examine performance with additional 
stimulus materials, and (c) examine the degree to v^ich the rewriting task 
correlates with free writing samples. 
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31. Gharavi, E, (1977). Admission of foreign graduate students: An 
analysis of judgoents by selected faculty and administrators at 
North Texas St^te University (Doctoral dissertation, North 
Texas State University, 1977). Dissertation Abstracts Inter- 
na tionp.l, 38, 1U8A. (University Microfilms No, 77-19, bbb) 



Purpose 

This study sought to determine, by use of the Judgment Analysis (JAN) 
technique, the admissions oolicies of faculty and administrators at North 
Texas State University for foreign graduate students. 



Method 



The subjects were 64 male Iranian graduate students randraily selected 
from among all 123 Iranian graduate students enrolled at North Texas State 
University during the spring semester of 1976. For each subject, data on 
the following variables were obtained from the graduate admissions office: 
age, marital status, school last attended, degrees held, years of English 
study, grade-poir.t average (GPA) in English courses, score on the five- 
part TOEFL [presumably obtained in an International administration!, 
monthly income. Graduate Record Examinations (GRE) Aptitude Te n verbal 
score (GRE-V), GRE quantitative scjre (GRE-Q), GRE total score, credit 
hours attempted during the first semester of enrollment, credit hours 
completed, cumulative first-semester GPA, and major field. [The GRE 
Aptitude Test is described briefly in Sunnaary No. A, America.! Association 
1971. J A profil? data sheet was constructed for each subject that 
Included information on all of the above variables. 

Twenty-eight judges (eight members of the central administration and 
20 departmental graduate advisers) examined the profile data sheets and 
rated each subject on a five-point scale: 1 - poor, 2 - below average, 3 
« average, 4 - above average, and 5 - outstanding. Each judge used 
whatev ;r information in the profile he or she thought most usefiil to judge 
the quality of the studevt, essentially simulating the type of rating that 
might be made in rendering an admissions decision [except that certain 
information used here, such as first-semester GPA, typically would not be 
available at the time of an admissions decision]. These ratings were 
analyzed using JAN, with the objective of determining which variables were 
commonly used by the judges, and how the varinbles were weighted, to 
assign ratings. In easerce, JAN analyzes the predictor eqiiation used by 
each Judge — i.e., the equation predicting the rating frwa the variables 
listed above — and then examines th« consistency of the predictor ecjjiatlons 
across judges. A stepwise multiple regression analysis was also conducted 
to determine the effect of each predictor variable on the {fredictive 
efficiency of the full regression model used in the JAN procedure. 
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Results 



Statistical analyii* of the different policies (i.e., prediction 
equations) used by the 28 judges showed that the policies of 20 of the 
judges could he classified into seven different types of policy, with each 
of the reaaining eight judges using a policy unique to the individual. 
Thus, although there w^s aoae consistency in the manner in which the 
judges aade their ratlniss, the judges did not use a single policy for 
judging the subjects' qualifications* The strongest predictor variables 
(TCJFL, GRE-V, GRE-Q, 6RE total, and f irst-senester CPA) showed high 
positive correlations with the judges* ratings. [Correlations with 
individual policies are given. HcHiever, correlations between the predictor 
variables and the ratings assigned 1^ the entire group of 28 judges are 
not.] Monthly income in U.S. dollars was ne|;&tively correlated with the 
judges' ratings. The aost freqtiently consider^ variable was the TOEFL 
score, which contributed significantly to the prediction equation for each 
of the seven types of policy mentioned above (i.e., ttu>8e associated with 
more than one judge each). 

JAN was found to be relial>le in identifying and analyzing rating 
admission policies. Also,. JAN showed that by attempting to group all 
judges into one judgmental policy, tt^ predictive efficiency of the system 
dropped dramatically from .76 to .32, a 44 percent loss in predictive 
efficiency. 



Conclusions 



It appears that, for the sample studied, (a) there was no single 
admissions policy for foreign graduate students; (b) while individual 
judges were reliable in tl»ir ratings, policies varied across judges and 
groups of judges; and (c) the most salient variables considered in the 
admissions process for foreign graduate applicants weie the TOEFL score, 
GRE~V, GRE~Q, GRE total, and first-semester GPA. It is suggested that in 
future studies judges be restricted to rating only applicants to their own 
departments, rather than all applicants. 
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32. Gzadman* H. L. » & Spolsky, B. (1975). Reduced redundancy testing: A 
progress report. In R. L. Jones & B. Spolsky (EdsOt Testing 
language proficiency . Washington* DC: Center for Applied 
Linguistics. 



Purpose 

This paper summarizes the results of several studies involving the 
noise test, a form of dictation test m which white noise is added to the 
stimulus nessager Of the research discussed in the paper, an exi^riment 
examining the relation of the noise test to TOEFL is eatphaslsed in this 
sianmary • 

Background 



The rationale underlying the noise test is that a decrease in redun- 
dancy of the message in a dictation tests such as that produced by back- 
ground noise, should result in a greater increase in the difficulty of the 
dictation test for a nonnatlve English speaker than for a native English 
speaker. Thus, differences between these groups stiould be accentuated, 
allowing greater precision of measurement. 

The original work with the noise test consisted of several experi- 
ments designed to develop the most effective test. The effects produced 
by varying the specific sentencas used, the signal to noise ratio, and 
other factors were examined. The final version of the test correlated .66 
with both an aural -comprehension test and an objective paper-and-pcncil 
test, and .51 ^±th an essay test. Other investigators have observed 
correlations in the .50s between the noise test and other [unspeclf iedl 
meastnres of English proficiency. 



Method 



In the experiment involving TOEFL, the subjects were 26 Saudi 
Arabian students in a speci:4l English training program at Indiana Univer- 
sity. The students were all given the noise test, the five-part TOEFL, 
the Ilyln Oral Interview, and the Grabal Oral Interview. 



Spolsky, B., Sigurd, B., Sako, M., Walker, E. , & Arteiburn, C. 
(1968). Preliminary studies in the development of techniques for testing 
overall second language proficiency. Language Learning , 18 , Special Issue 
No. 3, 79-101. 
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The noise test used here ifas a nultlple-choice version la which the 
subjects wre given five alternatives and told to chooee tht one that nost 
closely approxinated the sentence that was presented on tape with accoo- 
panying white noiae. The SO aentencea developed in tte original work 
cited above ware uaed. In the Ilyin Oral Interview, the subject la shown 
pictures and asked specific questions about thea. The Grabal Oral Inter- 
view is a test of free conversation; tht sid>Ject*s responses are rated on 
a nine-point scale for 10 categories two separate judges. 



Results and Conclusions 



The noise test correlated .75 with total TOEFL score, which was 
higher than its correlation with any ott^r test or any TOSFL subtest. All 
correlations involving the noise test were above .60, except the correla- 
tions with English Structure (.44) and Writing Ability (.33). Unexpect- 
edly, the noise test correlated .73 with TOEFL Vocabulary and .68 with 
Reading Comprehension. 

The Grabal Oral Interview correlated .73 with TOEFL total. Of 
correlations reported between this measure and tt^ TOEFL subtests, the 
correlation with Vocabulary was .71, but that with Writing Ability was 
only .17, and that with Reading Comprehension was .38. The correlation 
between the Ilyin Oral Interview and TOEFL was .54. 

Other reported results include a correlation of bet^en the noise 
test and the Ilyin Oral Interview and a correlatlv of .59 between the 
Ilyin and Grabal Oral Interviews. [Note that, as an Integrative paper, 
this report ie not designed to present all possible correlations.] 

The data indicate that the noise test was somewhat more highly 
related to a discrete item test, TOEFL, than to either of the oral inter- 
views, which are presiaaafoly more, functionally oriented. This may be due 
partly to the fact that the noise test used here was a multiple-choice 
type, so that It vas, in effect, a cross between a functional test and a 
discrete item test. In general, tl^ results suggest that the noise rest 
functioned well, providing a type of data that was not provided by the 
somewhat less structured interviews. 

The data from this experiment are general.ly in line with the results 
obtained in another study involving 25 Saudi Arabian students administered 
the noise test, the Cirabsl Oral Interview, and the TCffiFL. In that study 
the noise test correlated .66 with TOEFL total, .75 with TOEFL Listening 
Comprehension, and .79 with the Gtabal Oral Interview. 



Additional Discussion 



Results from subsequent research on the noise test not involving 
TOEFL are also discussed. In one study, both the multiple-choice and 
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original dictation version of tha nolle test were administered to 17 
native speakers of English ai^ 17 nonnatlve speakers. For each test, the 
top scores were earned by native speakers and the bottom scores by nonna- 
tlve speakers, showing that the test effectively discrlainates between 
these two groups. The two forms of the test correlated .89 for the 
nonnatlve speakers, indicating that both forms tend to measure the same 
thing. In an additional study with 71 nonnatlve English speakers, the 
standard (non-multlple-cholce) form of the test correlated .56 with the 
Indiana University placement examination, a test of English proficiency 
with several subtests. 

In general, the data show that the noise test clearly separates 
native from nonnatlve speakers; it correlates relatively well with other 
measures of language proficiency; and it discriminates well between weak 
and strong nonnatlve English speakers. Among the test's advantages is its 
ease of administration and scoring. 
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33. Cue, R.t & Holdavay, E. A. (1973). English proficiency tests as 
predictors of success in graduate studies in cniucation. Langu- 
age Learning , 23 , 89-103. 



Purpose 



This research studied the validity of TOEFL scores and interviews as 
predictors of grade-point average (GFA) among a group of Thai graduate 
students in education. Gain in TOEFL scores as a result of participation 
in an Intensive English language program was also assessed. 



Method 



The subjects were 123 Thai educators participating in 1967 throtigh 
1970 as graduate students in a one-year program in education at the 
University of Alberta. The areas of the subjects' specialisation were 
administration, supervision, academic curriculum, industrial arts, and 
guidance and counseling. The subjects ranged in age from 25 to 56 years; 
79 were males and 44 were females. 

The subjects were administered the five-part TOEFL twice. The 
first administration was in late June or early July after arrival at the 
University of Alberta; this admlnistratloa is here termed the "sumn^r 
TOEFL." Three months later the subjects were administered TOEFL again; 
this administration is termed the "fall TOEFL." In the intervening 
period, the subjects participated in an intensive English langui^e pro- 
gram. For each subject, the GFA wbs computed at the end of the one-year 
educational program. Interview scores were also collected by Canadian 
interviewers in Thailand prior to selection of the students for admission 
to the program in 1967 and 1968. The 1967 panel ratings were given as one 
global score, while 1968 panel ratings were subdivided into scores labeled 
"language fluency" and "potential for the Alberta program." 



Results 



Intercorrelatlons amo<*.g TOEFL subtest scores were craputed, with 
subjects pooled across years. Correlations among subscores ranged from 
.62 to .82; all of these correlations were significant at the .01 level. 
The correlations within each of the years showed more variation, the 
extremes ranging from .37 to .89. For data from each of the separate 
years, the highest correlations were between English Structure and Writing 
Ability {r^> .70) and between Vocabulary and Writing Ability (r > .65). A 



103 



-98 



twf actor factor analysis with variaax rotations was conducted for suamer 
and fall TOEFL scores, with data pooled acroas years. The first factor 
identified waa aost closely associated irlth Engliah Structure and Writing 
Ability, the aecond factor waa closely dissociated with Listening Compre- 
hension and vltb Reading Coftprehenalon. A three-factor analysis identifi- 
ed the saae first factor; a second factor, on «^ieh Reading Comprehension 
and Vocabulary subscores showed the highest loadings; and a third factor, 
on which Liatening Comprehension showed the highest lotting. 

The subjects* grades ranged from 1 to 9, with S considered a passing 
mark. Overall, the average CPA of the subjects i»a 6.6, with the range 
extending from 6.4 to 6.9 over Individual years. Mean scores on the 
summer TOEFL ranged from 374.3 to 452.5, with an overall mean of 424.6. 
Fall TOEFL score means ranged from 391.7 to 480.2, with an overall mean of 
447.8 across /ears. For each year, fall TOEFL scores wre 20 to 30 points 
higher than summer TOEFL scores. Over the four years, Intercorrelatlons 
among summer TOEFL aubscores and fall TOEFL subscores ranged from .28 to 
.73. For data pooled across years, summer and fall TOEFL scores correlat- 
ed .49 and .59 with final CPA; these correlations were significant at the 
.01 level. 

Table 1, based on pooled data, summarizes the TOEFL summer and 
fall Bubscore means, the correlation of suamer and fall subscores with 
CPA, and the level of significance of the correlations. 

Table I 

Mean and Correlation of Each TOEFL Subtest with CPA 



GPA- 
Subtest 



Subtest 


Time 


Mean 


Correlation 


Listening 


Summer 


43.6 


0.38 


Comprehension 


Fall 


46.8 


0,52 


English 


Summer 


44.6 


0,51 


Structure 


Fall 


47.2 


0,55 


Vocabulary 


Sumror 


J9.6 


0.34 




Fall 


42.4 


0.48 


Reading 


Summer 


42.0 


0.38 


Comprehension 


Fall 


43.1 


0.51 


Writing 


Summer 


42.5 


0.51 


Ability 


Fall 


44.4 


0,53 



Note. All correlations are significant at the .01 level. 
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Stepwiie multiple regression analysis was used to predict GPA 
froiB stmmer TOEFL subscores for each progrsa year and for progm years 
combined. The results did not consistently identify any single TOEFL 
subscore as aore important than any other subscore in predicting GPA. The 
addition of interview panel ratings as pradictor variables in r^reasion 
analyses for 1967 and 1968 students led to contrasting results. The 1967 
interview panel rating did not add significantly to prediction of GPA. In 
contrast, the language fluency rating obtain^ in the 1968 Interview by 
itself accounted for 41. S percent of final GFA variance; this rating was 
far nore important in predicting final GPA than i^re the suamer TOEFL 
subscores. 



Conclusions 



The high intercorrelations among TOEFL subscores were consistent with 
the results of previous studies. The clustering of Vocabulary, English 
Structure, and Writing Ability into one factor suggested that TOEFL could 
be reduced In length wltlwut loss of predictive power. 

The contents of some Listening Comprehension items might be criti- 
cized, as the use of narratives concerning disasters and unhapplness could 
affect examinees* responsiveness. In addition, the TOEFL scores obtained 
in the present study might have been affected by the organizational 
climate of rest taking: the examinees were aware that their TOEFL scores 
would not be used for their selection into a program, and this awareness 
ioay have had an Indeterminate effect on their performance. 

Gains In TOEFL scores across summer and fall adnlnlstratlons were 
encouraging and vere greatest for those students who needed the summer 
language program the most. Nineteen other students » however, earned lower 
TOEFL 'scores In the fall than in the susmer. Rea»>ns for these declines 
were not apparent. 

A number of factors may constrain the generalizabllity of the results 
obtained, including differences in Thai student cohorts participating in 
the study and the grading standards of the University of Alberta. Use of 
interview panel ratings of English proficiency as adjuncts to TOEFL 
deserves further examination. 
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34. Hale, G. A*, Angclls. P. J*, & Thibodeau, U A* (1980)* Effects of 
item discloeure on TOEFL jwrforttance (TOEFL Research Rep« No, 8; 
ETS Research Rep, No« 80-340 Frlnceton, NJ: Educational 
Testing Service* 

Purpose 

Following several TOEFL administrations each :^ar» tl^ test forms 
used at those administrations are disclosed, or made public. Items fr<xn 
nondisclosed tests are reused in the Institutional Testing Program» and if 
reuse of disclosed it^s for this program were also to be considered, it 
would be important to know how TOEFL performance would be affected if 
candidates were to have access to some of the items before a test adminis- 
tration* 

To address this issue experimentally, specially constructed TOEFL 
forms, called ''disclosed forms,** were made available to foreign students 
in intensive English language programs. Later the students wre adminis- 
tered a special TOEFL consisting of items from those forms and a lOEFL 
with all new items. Superior performance on the former test would indi- 
cate a diaclosure effect. 

If students must cover a large numt^r of disclosed forms in order to 
be exposed to all it^s that will appear on a later TOEFL, they should be 
less likely to benefit from having the disclosed forms available than if 
they need only cover a small nunber of forms* To test this hypothesis, 
items to appear on the special TOEFL were spread through six disclosed 
forms for students in some institutions, and through 12 forms for students 
in other institutions. 

It was assumed that, in reality, disclosed TOEFL tests are used in 
TOEFL preparation courses. Thus, to simulate a test-preparation situa- 
tion, the students discussed a portion of the disclosed forms in class. 



Method 



The subjects were foreign students in intensive English language 
programs at 20 U.S. universities; the final sample consisted of 668 
subjects who took all tests. 

Two three-part TOEFLs constructed from retired operational items 
comprised the ""posttests" given at the end of the study. The posttests 
were administered one week apart at most institutions. A TOEFL -pretest" 
was also given at the outlet of the study, to be used as a covariate for 
statistical control. Tests ipere administered by the participating insti- 
tutions according to ETS guidelines for Institutional administrations. 
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The "disclosed forms" made available during the study were also 
constructed froa retired operational Iteas and were similar to operational 
TOEFLs except that each correct answer was starred. Interspersed through 
the 4i*<^^OB6<i forms were iteas that would appear on one of the posttests* 
termed the "disclosed posttest"; the "undisclosed posttest," in contrast, 
contained all new items* 

The disclosed forms were distributed to students in 16 Institutions, 
and the subjects were told of the upcoaing special TOEFL, In which they 
would encounter items from the disclosed forms. The subjects vtre in- 
formed that their test scores would have no bearing on their academi 
standing or their admissions status. 

Subjects in eight institutions were given six forms, and in eight 
other institutions t 12 forms. The forms were available for a n^ian of 29 
days. Within each of these groups, subjects in four institutions received 
the "disclosed posttest" first, and subjects in four other institutions 
received the "undisclosed posttest" first. Tapes of the audio portions of 
the Listening Comprehension sections of ttese forms could be obtained from 
a central facility for listening to at that facility or at home. 
«> 

Subjects in four additional control institutions were given only the 
pretest and two posttests, to check on the relative difficulty of these 
tests. [The data showed only a small difference between posttests, and 
the counterbalancing in order of tests wai. expected to correct for this 
difference; results for these four control institutions are not presented 
here. ] 

The Items from the disclosed forms that were discussed in class 
comprised the equivalent of one fuIV TOEFL, although not all items were 
taken from the same form, and approximately five hours was devoted to the 
class discussion. 

S 

Results and Conclusions 



A significant disclosure effect was observed, as scores were 4.6 
percentage points higher on the disclosed posttest (55.9 percent correct) 
than on the undisclosed posttest (51.3 percent). (Scaled scores cannot be 
provided for these specially constructed tests.) Analysis of covariance, 
with the pretest as a covariate, showed that this effect could not be 
attributed to group differences in English proficiency. It appears, then, 
that when TOEFL items were made available for a few weeks before the 
administration of a test containing tlu>se items, the subjects tended to 
study and recall many of those specific Items. 

For items discussed in class, the disclostire effect, examined only 
for the condition involving six disclosed forms, was a full 11.8 percent- 
age points. Yet an effect was shown for items not discussed in class as 
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well— 4.4 pareentage point* for ttudents given aix ditcloied fonui and 
2.0 points for those given 12 forms; all of these effects were signifi- 
cant* To soae extent, then, the students studied the disclosed foras on 
their own Initiative. Thla conclualon is reinforced hy data froa a 
questionnaire. In which the aubjects reported averaging between three and 
four hours of listening to the tapes and between five and six hours of 
reading the disclosed forms. Thus, It appears that students are motivated 
to study disclosed iteas that will be encountered on a later TOEFL, even a 
nonoperatlonal test. 

The disclosure effect wae greater when the itesis to appear on tl^ 
postteat were spread tlurough six disclosed foms (6*3 percentage points) 
rather than 12 foras (2«9 percentage points), nie data thus support the 
hypothesis stated above: If students must cover a relatively large number 
of disclosed foras in order to he exposed to all items that will appear on 
a later test, they are less likely to benefit from the opportunity to 
study disclosed forms than If ttey need cover a smaller nust^r of forms. 
This result has important implications regarding possible reuse of dis-* 
closed items in Institutional TOEFLs* As the pool of disclosed TOEFL 
forms increases over tlme» tYiexe stould be a decrease in the proportion of 
items in the pool that students can study and remember and, thus, a 
decrease xn the effect due to disclosures 
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35. Harv«y, M. J. (1979). Acadeadc achievement as predicted by the Test 
of English aa a Foreign Language . Unpublished tnaster's thesis, 
For timid State Univeralty, Portland, OR. 



Purpose 



This study evaluated TOEFL as a predictor of foreign students* 
academic achievement during their firet two terns at Portland State 
University, with attention given to the influence of graduate versus 
undergraduate status and college of enrollment within the university. The 
initial sections of the thesis review measurement and otter problems in 
devising prediction schemes for foreign students* achievement on the basis 
of language proficiency information and aci^emic aptitude information. 
Various studies Involving the relation of TOEFL to other proficiency tests 
and to collide grades are reviewed. 



Method 



The subjects were 78 foreign Ptiidents at Portland State University 
who met the criteria specified below. Scores on the three-part TOEFL 
I presumably earned In laterr.ational or Special Center administrations] and 
grades were obtained from administrative offices at Portland State Univer- 
sity. From the total population available, students were eliminated on 
the following bases, to yield the final sample of 78 subjects: (a) a time 
span of more than six months between test date and enrollment; (b) TOEFL 
score based on the earlier five-part examination; (c) TOEFL score below 
500 I note, however, that analyses reported below involve subjects with 
TOEFL scores below 500); (d) concurrent enrollment in an intensive English 
course during the first two terms at Portland State University; (e) 
enrollment in less than eight hours of academic work at the beginning of 
the first or second term of study; and (f) transfer of previous toursework 
from another American school to Portland State University. 



Results 



TOEFL total scores correlated .18 with grade point-average (GPA) for 
these foreign students < .05). Correlations of GPA with the Listening 
Comprehension section and with the Reading Comprehension and Vocabulary 
section of TOEFL were both .15; the correlation of GPA with the Structure 
and Written Expression section was .03. Graduate students who scored 
below 500 on TOEFL or between 500 and 550 earned higher average grades 
than did undergraduates in the same TOEFL score ranges. Undergraduates 
earned a higher mean GPA "ban did graduates in the TOEFL score range 550 
and above. 



109 



Exm&inatlon of relationships betveea TOEFL scores and grades within 
college of enrollment was based on tabulation of average grades in the 
three TOEFL score ranges: below 500, 500*^550, and above 550 • The colleges 
Investigated were arts and letters » sclencst social science^ health e^id 
physical education » and business administration. Interpretation of the 
resulting data was inconduii^^r ^th trends going in 'opposite directions 
for the lowest two TOEFL scor^ levels* CPA was higher for students with 
TOEFL scores above 550 than for students with TOEFL scores in the 50O--55O 
range. This relationship was evidenced for four of the five colleges. In 
contrast to this pattern, students in business administration earned lower 
grades if they scored above 550 on TOEFL. 



Conclusions 



The present data demonstrated tliat TOEFL scores were of limited value 
in predicting CPAjt^ (Mention is made of a lack of consistent correlation 
between TOEFL total and TOEFL part scores according to college of enroll'- 
ment; however, the relevant correlation coefficients are not presented in 
the thesis*] A redesign of TOEFL might be considered, with the various 
TOEFL test sections bearing a XBore natural relation to everyday academic 
tasks than is currently the case. A measure of writing la needed and, in 
general, the importance of productive skills in assessing English profi** 
ciency must be considered. It is sieges ted, however, that revision of 
TOEFL might not necessarily lead to its Improvement as a predictor of 
college grades, since the importance of English language proficiency to 
academic achievement may have been exaggerated in the literature. 
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36. Hassan, K. I. (1982). The corf lation betwen performance on the 
WAIS-R Vocabulary aubteat and the TOEFL . Uupublishc'*. master's 
theals, Washington University, St. Louis, HO. 



Purpose 



This study coapared interrelationships in perfomance on the five- 
part TOEFL, the Vocabulary subtest of the Wechsler Adult Intelligence 
Scale-Revised (WAIS-R), and college grade-point average (CPA) among 30 
male Arabic students at Washington University in Missouri. Three hypo- 
theses are set forth: (a) There is a significant relationship between 
TOEFL score and the score on the Vocabulary subtest of the WAIS'^R; (b) 
There is a significant correlation between TOEFL score and semester 
grade-point average (CPA); and (c) There is a significant crrelation 
between the score on the Vocabulary subtest of the WAIS-R and semester 
grade-point average (CPA). 



Method 



The 30 subjects under study were enrolled as full-time undergraduate 
students at Washington University; all were male and had Arabic as the 
native language. Subjects ranged in age from 18 to 29 years with a median 
age of 21.5 years. Student records indicated that they had studied 
English for a period ranging from three to six years. The tests used in 
the study were the five-part TOEFL and the Vocabulary subtest of *he 
WAIS-R. In the WAIS-R Vocabulary subtest there are 35 words to be defln* 
arranged in order of Increasing difficulty. For each word the subject : 3 
to state what the word means. This test was administered to the subjects 
In the spring semester of 1982. TOEFL scores and semester CPA were 
compiled for the subjects; the subjects had taken TOEFL prior to the 
study I presumably in International or Special Center administrations}. 
Pearson product-moment correlations were computed for pairs of measures. 
Multiple regression analysis was used to predict swaester CPA frcHu TOEFL 
scores and from WAIS-R Vocabulary scores. 



Results and Conclusions 



The means and SDs for the three measures were WAIS-R Vocabulary 
(6.66, 1.68); TOEFLI 459.1.0, 58.08); and CPA (2.45, .35). The WAIS-R 
Vocabulary acore correlated .25 with TOEFL total score; this relationship 
was not itatistically significant. The TOEFL score correlated .45 with 
semester CPA; this relationship was significant at the .01 level. The 
WAIS-R Vocabulary subtest score correlated .27 with semester CPA; this 
relationship was not statistically significant. 
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The regression analyses yielded the follovlng prediction equations: 
CPA - .056 (Vocabulary) + Z.OS, with standard error of estimate - .34; and 
GPA - .0027 (TOEFL) -f- 1.211» with standard error of estimate .80. [No 
regression analysis is reported including both TOEFL score and Vocabulary 
score as pri^ictors of 

The findings are brsed on a small sample of students , and caution is 
^:<rged in attempting to generall£^^ «:he findings to other tmiversities* 
Further research involving non-Ar^abic speaking foreign atudtf'at^ is sug-* 
gested. 
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37. Hell, D. K., 6 iQeamoni, L. M. (1974). Assegiaent of the proficiency 
In the uge and understanding of EnjgllBh by foreign studenta aa 
y.c«snred by the Teat of Engllih as a Foreign language (Report 
No. RR-350)^ Urbana, tL: University of Illinois. (ERIC 
Docuaent Reproduction Service Mo. ED 093 948). 



Purpose 



This study Investigated the prediction of foreign graduate students » 
Rcadenlc achievement using scores on tests of English language proficien- 
cy. The specific obj xtlves of the study were: 

(a) to d&tenalne the predictive validity of TOEFL, with first- or 
seconi* semester graduate grade-point average (CPA) serving as the 
criterion; 

(b) to exaaine the predictive validity of the English Placement 
Examinatica (EPE) developed at the University of Illinois at 
Urbana-Champaign lUIUCK using first- and second-semester gradu- 
ate CPA a» the criterion; 

(c) to exs'Alae the concurrent validity of TOEFL and the EPE; 

(d) to determine the predictive validity of TOEFL and the EPE. using 
grade in a remedial English course as the criterion; and 

(e) to estimate the degree of change in total TOEFL score resulting 
from living in an English-speaking country and taking a remedial 
English courss for one semester. 

Method 



The subjects were 148 incasing graduate foreign students accepted for 
admission to UIUC in the fall of 1970. Scores on the five-part TOEFL, 
(presumably administered in the Institutional Testing Program] were 
obtained from admissions office records. A second set of TOEFL scores was 
also obtained, based on an administration of TOEFL upon the subjects* 
arrival At the university in September 1970. Students scoring in the 
range of 480-569 on the preadmission TOEFL were required to take the EPE 
test administered by the Division of English as a Second Language (ESL). 
The EPE consists of four parts. The Structure section tests recognition 
of English grammar and sentence strwrture In writing. Tha Aural Compre- 
hension section tests understanding of spoken English at normal speed. 
The Original Composition section tests ability to write on an assigned but 
familiar topic for which an outline has been provided. The Pronunciation 
section tests ability to understand and be understood orally. 
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The results of these exsBlnatloiui were used to place students in one 
of five levels of ESL courses* Those scoring aoove 569 on TOEFL were 
placed In one of the two highest-level ESL courses. Students scoring 
below 480 usually were not adaitted to the university. The nuBibers of 
subjects entering into different analyses in the study varied; 89 of the 
original 148 subjects were enrolled and coapleted coursetiork in the ESL 
classes. 



Besults 



Total GPA in either the first or second sesaester of study was found 
to correlate significantly with TOEFL scores obtain^ before and upon 
arrival at the university} correlations for part and total TOEFL scores 
ranged from .16 to .39. Higher correlations with TOEFL scores ^re 
observed for second-seaester GPA than for first-semester GPA. TOEFL 
scores obtained prior to admission correlated slightly higher with GPA 
than did TOEFL scores obtained at the tine of admission. 

Correlations between (a) EPE part and total scores and (b) first- and 
g€Cond-s eme»te r <gPA wer« tmch lower -than t4»e corraspondiag 
between TOEFL and GPA; the foraer correlations ranged In value from **^04 

to al2a 

TOEFL score (based on postadmlsslon testing) vas not significantly 
correlated %ilth the subjects' grades In their ESL classes , and EPE scores 
also generally failed to correlate significantly vlth ESL grades* Of 
grades received In the three aost heavily attended SSL classes ^ the grade 
In the lowest class was significantly correlated with GPA earned In the 
second but not the first semester; the reverse was true for the next 
higher level. For the third most l^avlly attended ESL class (the second 
highest level in the sequence)* ESL grade was significantly associated 
%dth both first- and second**se9ester GPA# 

Results of a regression analysis indicated that the grade in each of 
the tvo lowest ESL clasfes was significantly predicted by coabined EPE and 
TOEFL scores. In the case of one of these classes » the second-*level ESL 
class, EPE score significantly iaproved prediction of ESL grade when added 
to TOEFL as a predictor variable in regression analysis. 



Conclusions 



The predictive validity of TOEFL for foreign students appears to be 
similar to the predictive validity of admissions test scores for native 
Americans. This result siiggests that academic success may be no more 
predictable for foreign students than for native Africans* Correlations 
between TOEFL part scores and GPA might be Improved if part scores more 
adequately reflected skills requlri^ in collide work. 
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38. Rlllnan, R. E. (1973) • A correlational study of selected vocal- 
verbal behavlora and the Test of English as a Second [sic] 
Language (TOEFL) (Doctoral dissertation, Pennsylvania State 
University, 1972). Dlasertation Abstracts International , 34 , 
1397A. (University Mtcrofilas No. 73-20, 090) 



Fttrpose 



This study attempted to validate TOEFL as a neasure of the ability 
to encode oral English by using vocal-verbal behaviors as criterion 
aeasures. It also atteapted to test the hypothesis that nonnatlve English 
speakers exhibit nore proficiency when speaking about experiences in their 
home countries than when speaking about experiences in the United States. 



Method 



The subjects for the study were 124 undergraduate and graduate 
students enrolled at the Univ ersity of Wisconsin during the fall semester 
or 1971. l>ue~to attrition and technical TTifflcuIt data are presented 
for only 47 students. A 1968 form of the five-section TOEFL was adminis- 
tered as part of the study. Also, the five vocal-verbal behaviors de- 
scribed below were elicited during a structured oral interview. The 
interview Included questions about life in the student's home country and 
life in the United States. Each interview wss recorded on audio tape and 
transcribed. The subject's first 100 words regarding life in the home 
country and first 100 words regarding life in the United States were 
scored separately for each behavior. 

The five behaviors chosen were (a) amount of silent pause time 
(Including any pause greater than .25 seconds as determined by a recorder, 
which transmitted speech onto continuous graph paper); (b) ntmber of 
verbalised pauses, such as "uh" and 'Veil," as counted by five judges; (c) 
type/token ratio (the number of different words divided by the total 
number of words); (d) average number of words five judges were unable to 
identify; and (e) average overall English language proficiency (ELP) 
rating as assigned by 10 judges on a seven-point bipolar scale, with 
"native" and "nonnatlve" serving as the poles. 

In addition, two indirect measures of English proficiency were 
administered to the subjects. These trere the Indirect Measure of Oral 
Output (IMOP) and the Holtsman Stress-unstress rest. The IMOP is a 
20-mlnute listening test in which the examinee classifies speakers as 
either native or nonnatlve. The Holtman Stress-unstress test requires 
examinees to read aloud a list of 31 words; each response is scored for 
correct placement of stress. 

Intercorrelatlons were computed among TOEFL scores, the five vocal- 
verbal criteria corresponding to the home country and U.S. portions of 
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the oral interview, and the two additional measures. In addition, an 
analysis of variance vaa performed to determine if the scores from the 
home country and U.S. sections of the interview could be merged. Inter- 
rater reliabilities %rere determined where appropriate. 



Results 



The mean TOEFL score for the group was 475, %rtiich is slightly below 
the average of all examinees taking TOEFL in International and Special 
Center administrations. The standard deviation, 83, was five points 
greater than^ average. As noted above, the three variables that employed 
more than one Judge were verbalized pauses, unidentified words, and 
overall English language proficiency. Reliabilities of the ratings for 
verbalized pauses (.87) and for English language proficiency (.90) were 
regarded as acceptable, as both were greater than .80. However, the 
reliability of unidentified wrd counts (.29) was not acceptable* 

The analysis of variance showed that the means of scores on the five 
vocal-verbal criteria for the home-country context did not differ signifi- 
cantly from the means for the U.S. context. However, analysis of the 
intercorrelations between the five vocal-verbal behaviors and two indirect 
measures showed that some iBignificant relationships did exist between home 
country and U.S. speech behavior. Verbalized pauses, unidentified words, 
and English language proficiency each yielded correlations above .60 
between the two speech contexts. Verbalized pause counts and unidentified 
words yielded the highest correlations with English language ratings. 

The correlations between TOEFL total score and the five direct and 
two Indirect criterion measures are presented in Table 1. It was hypothe- 
sized that the TOEFL score would show a negative correlation between 
number of silent pauses, verbalized pauses, and unidentified words. 
Similarly, it was hypothesized that TOEFL score would correlate positively 
with ELP ratings and type/token ratio. It was also hypothesized that 
TOEFL score would show a positive correlation with the two additional 
me asures. 
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Correlations between TOEFL and Seven Language Measures 



V xable 



Home Country 
Context 



U.S. 

Context 



Silent pause 
Verbalised pause 
Type/Token ratio 
Unidentified words 
ELP ratings 
BlOP 

Stress-unstress test 



.07 

.51* 

.14 

.27 

.56* 



-.23 
-.29 
-.42* 
-.13 
.36 



.52* 
.35 



*SignifiCdnt at the .01 level. 



As indicated in the table, few correlations significant at the .01 
level were obtained with this sample. TOEFL correlated significantly %^th 
(a) verbalized pauses In the home-country context , (b) type/ token ratio in 
the U.S. context, (c) English language proficiency in the home^country 
context, and (d) the IMOP test of ability to recognize the pronunciation 
of a native speaker. Thus, only four of the 12 comparisons yielded 
statistically significant correlations. For each of the five direct 
measures, the difference between correlations in the home-country context 
and the U.S. context was examined statistically; the difference was 
significant for three of the direct measures. However, the general 
direction of the differences favored a stronger relationship between TOEFL 
scores and the ability to use English when speaking about home-country 
experiences. 



Foreign students with high TOEFL scores may exhibit fewer verbalized 
pauses and greater English language proficiency when talking about what 
they have experienced in their home countries than when talking about what 
they have experienced In the United States. Thus, the ability of TOEFL to 
predict vocal^verbal behaviors may depend on the context of those behav 
iors. 

It is possible that the generally superior relationship between TOEFL 
scores and ability to encode in the home-country context is due to greater 
familiarity with the home-country context. The learners, being familiar 
with what is an appropriate response to questions in that context, may be 
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able to utilice their language proficiency to aoniCor their output. Lack 
of faailiarity with the nature of expected responaes in the U.S. context 
nay result in the aonitor either not being invoked or being underuaed. 
Thua, a reduced correlation between verbal output and TOEFL acore i»uld be 
expected. Vhile the data do not i^ovide atrong aupport for a relationabip 
betneen TOEFL icore and real language uae, the precise relationship may be 
iaposaible to establish until research yielda more Information on the 
vocal-verbal behavior of nonnative speakers of English. The absence of a 
correlation between pauae-time and ELF ratinga suggests that appropriate 
pause-time paraaetera for nonnative apeakera ne^ furtter investigation. 
The ssme may be true of other variablea included in thia study. 
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39. Hlnofotla, F. B. (1980). Close «• &n alternative method of ESL 
placeaent and proficiency teating. In J. W. Oiler, Jr., & K. 
Perkina (Eda*), Reaearch in language teating . Rowley, MA: 
Kewbury Houae. 



Furpoae 



Thia article, baaed on the author* a diaaertation, addreaaed two baaic 
quest iona relating to close teating: (a) Can a cloze teat aerve aa a 
surrogate meaaure of Sngliah language akilla without algniflcant loaa of 
information and (b) Should close teat a be scored by accepting only the 
exact word deleted, or ahotild otfwr graBusatical and contextually aultable 
substitutes be accepted aa veil? 



Method 



Inconing foreign atudenta repreaenting a variety of native language 
backgrounds served at aubjectn. 411 were enrolled at the Center for 
English as a Second Language (CESL) at Southern Illinoia University. A 
total of 107 aubjecta took both a close teat and the CESL Placeaent Test 
at the beginning of two consecutive aix-tMsek terma during the sumaer of 
1976. (The CESL Placeneot Teat ia a three-part test yielding a total 
score and aubacores for liatening conprehenaion, atructure, and reading. 
Scores are grouped within aix proficiency levela.) In addition, 52 of 
the subject a, who acored at level three (of six) or above on the CESL 
Placement Test, also took the five-part TOEFL at the university testitig 
service. 

The cloze teat employed a 427-word passage, adapted from an inter- 
mediate level text in English aa a aecond language, that described differ- 
ent means of tranaportation uaed for long distance travel. Every seventh 
word was deleted from the paaaage to yield a total of 50 blanks. The 
subjects were given 30 minutes to complete the close passage. First, 
scores were obtained by accepting as correct only tt» exact word that had 
been deleted (close-exact method). Then, the papers were rescored by 
counting aa correct responses that were grammatically and contextually 
acceptable (cloze-acceptable method). Correlations vere then obtained 
between section and total acorea on the two tests of English language 
proficiency and scores on the cloze test as computed by the two methods. 



Results 



As shown in Table 1, total acores on the CESL Placeaent Test and 
TOEFL were highly correlated with acores on the close teat obtained under 
both acorlng aethods. Also, reading acores on both tests correlated more 
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highly with the cloze test than did scores on any other aubtest. (The 
correlations were all significant beyond the .005 probability level.) 
However^ the correlations may have been attenuated due to the lack of 
reliability of the cloze paaaage {.bl for doze'-exact ecore and »8S for 
cloze-*acceptable score) » which was found to be very hard for this group of 
students* 

Cloze-exact and cloze-acceptable scores did not show significantly 
different correlations with the CESL Placement Test ( «8U and .84) but did 
show significantly different correlations with TOEFL (»71 and «79). 
Stlllt the correlation between these two scores was •97# 



Table 1 

Correlations of Cloze Test Scores with Total and 
Subtest Scores on the Criterion Measures 



Cloze**exact Cloze**acceptable 





Score 


Score 


N 


Total CESL Placement Test 


.80 


.84 


107 


CESL Listening Comprehension 


.71 


.73 


107 


CESL Structure 


.63 


.69 


107 


CESL Reading 


.80 


.80 


107 


Total TOEFL 


.71 


.79 


32 


TOEFL Listening Comprehension 


.47 


.51 


52 


TOEFL English Structure 


.51 


.38 


32 


TOEFL Vocabulary 


.59 


.62 


52 


TOEFL Reading Comprehension 


.68 


.77 


52 


TOEFL Writing Ability 


.55 


.64 


52 



Conclusions 



The high correlations obtained in this study warrant the cautious 
use of the cloze test as a measure of English language proficiency. 
Although the two scoring methods apparently cause subjects to rank order 
in the same manner, the cloze-acceptable method shows nuch greater vari- 
ance and greater reliability. The cloze-exact method is less time consum- 
Ing, but the cloze-acceptable method is more reliable and therefore 
preferable. 



ERIC 



120 



Di, 13 



40« Hoaburgt !• J* (1979)^ TOEFL and GPA: An analysis of corralatlona. 

In R» Silveratein (Ed.)» Proceedings of the Third International 
Conference on Frontiers in Language Proficiency and Domln«ace 
Testing s Occasional Papers on Llnguisticsi No. 6» Carbondalet 
IL: Soutimm IllinoiiirtJniversity. 



Purpose 



This paper discusses advice rendered the advisory board of the Office 
of International Education at Southern Illinois University (SIU) on use of 
TOEFL. 1979 1 SIU required a minimum TOEFL score of SSO for foreign 

applicants to graduate programs and a minimum TOEFL score of 525 for 
foreign applicants to undergraduate programs. At SIU there were contrast** 
ing vievs regarding use of cutoff scores in admissions. Some expressed 
concern that the cutoff scores used for admission^ particularly graduate 
admission, were set too high for students %dth good credentials. Others 
expressed concern that minimal TOEFL score requirements were not adequate 
indicators of linguistic ability to perform in certain programs of gradu-* 
ate study. In considering possible changes in the use of TOEFL scores in 
graduate admissions , three questions were addressed: (1) How are TOEFL 
scores used? (2) What are some of the problems with this process? 
(3) What should be done to Improve the current system? 



Discussion 



The 1978 TOEFL manual^ recommends that use of TOEFL in admissions 
should not include the prediction of scores on academic achievement 
indices » such as grade-point average (GPA). TOEFL is a test of profi- 
ciency in English as a second language and not an achievement test. There 
is no systematic or intentional connection between TOEFL test demands and 
academic materials that foreign students may have mastered earlier. 

The TOEFL manual also warns against using cutoff scores on TOEFL. 
This recommendation is supported by two factors. First, TOEFL scores are 
legitimate indicators of the relative standing of examinees with regard to 
English language skills; these scores are not Intended to convey informa- 
tion about candidates* English language skills in particular situations at 
particular InstltutlonSt Second, because there Is some degree of error in 
measurement y use of cutoff scores in adotesions can lead to mistakes in 
application of a TOEFL cutoff score in admissions decisions. 

The Inapproprlateness of TOEFL score as a predictor of achievement is 
demonstrated in a study at SIU that Investigated the correlation between 
TOEFL score and GPA in each of two groups of foreign graduate students 
(]*8 - 134 and 54). [The source of TOEFL scores, and whether they were 
derived from the three-»part or five-part TOEFL, are not indicated.] 



Educational Testing Service (1978). TOEFL test and score manual , 
Princeton, NJ: Author. 
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The first group had coapleted their program of ttudiei while the second 
group had not. The correlatiooi I)«t«e0s TOFFL scores and GPAa for the t«o 
groups were -.07 and .22, respectively. For both groups combined, the 
correlation between TOEFL score and CPA was .11 • Rest "s of a regression 
analysis indicated that there \mm aot a significant di. erence in GPA for 
the two groups; both groups de«on«trate<l high aean GPAa, of 3.58 and 3.39, 
respectively. It is concluded th«t numerous factors could affect achieve- 
ment that might not be reflected in adiaiasiona test scores. 

Guidelines for using TOEFL scores cited in ,ae TOEFL manual are 
discussed. (These are essentially the guidelines that are listed in the 
summary of the 1981 manual (Sumiaary No. 28, Edxscational Testing Service, 
1981) summarized elsewhere in this collection*] 
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41. HosXey, D« (1978) • Performance differences of foreign stxidents on 
the TOEFL, TESOL Quarterly , 12, 99-100. 



Ptirpose 



Among foreign students in sn Intensive English language prograoi, 
differences in TOEFL^ scores as a function of such variables as country of 
origin ^nd sex were exmained* 



Method 



The sid>jects twre 147 foreign students in tte Center for English as a 
Second Language (CESL) at the University of Arizona. The subjects in the 
sample comprised 28 percent of tlm total nimber of stivients in the CESL 
program and were drawn from 19 different cotmtrie^. For data analysis, 
the subjects were placed into six groups: ^xico, Saudi Arnbia, Libya, 
Venezuela, and Japan (the five most common countries of origin), and 
others. Scores on the five-pare TOEFL were available (preswiably obtained 
through International test administrations*] 



Results and Conclusions 



Analysis of variance of tt^ TOEFL scores shwed the effect of country 
to be significant » with scores of Mexicen subjects being the highest and 
significantly different (via post-hoc tests) from those of Saudi Arabian 
and Libyan subjects* which wsre the lowest. 

Differences among the TOEFL subtests were also significant* with the 
mean score on the Writing Ability subtest higher than scores on the other 
subtests. The interaction between subtests and country of origin was also 
significant, with a post-hoc test showing that the Listening Comprehension 
and Vocabulary sections contribute most to the superior performance of 
the Mexican subjects. 

A comparison between 38 students who had been in the CESL program and 
H9 new students showed that TOEFL scores were signi.^icantly higher for the 
new students. The difference between males and foaales was not signifi- 
cant. These data should help to identify variables that can affect TOEFL 
performance and provide hypotheses for further study. 
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42. Hosley, D. , & Meredith, K, (1979). Inter- and intra-test correlates 
of the TOEFL. TESOL Quarterly , 13, 209-217. 

Purpose 

The constructs measured by TOEFL and the nature of language profi- 
ciency were exaalned using data obtained from students enrolled in an 
intensive English program. 

Method 

One hundred sixty-nine students enrolled at tte Center for English as 
a Second Language at the University of Arizona had scores on the five- 
section TOEFL [presumably administered via International or Special Center 
programs I and the Comprehensive English Language Test for Speakers of 
English as a Second Language (CELT). The CELT consists of three sections: 
Listening, Vocabulary, and Structure. The Listening section contains 
three parts. In the first part, the examinee hears a short question and 
then selects, from four alternatives, the correct answer to the question. 
In the second part, the examinee hears a statraieat and clK>oses, from four 
printed statements, the one closest in meaning to the one heard. In the 
third part, the examinee hears a short dialogue between two speakers and a 
question posed by a third sneaker and oust select, from four printed 
alternatives, the correct answer to the question. The Vocabxilary section 
contains two parts. For each it^i in the first part, the examinee sees a 
sentence with a deleted word and imist select, from four alternatives, the 
deleted word. For each item in the second part, the examinee reads a 
short phrase and then selects, from four alternatives r the word described 
by the phrase. For each item in the Stricture section, the examinee reads 
a short conversation between two persons, with a word or phrase deleted 
from the last sentence; the examinee must select, from four alternatives, 
the deleted word or phrase, [description paraphrased from a recent CELT 
manual ] 

Correlations between the five TOEFL sections were determined and a 
factor analysis was performed. The subjects were also divided into three 
proficiency groups on the basis of CEIT scores and teacher recojmaenda- 
tions. The correlation between TOEFL fccore and group placement was then 
determined, as were the correlations between sxibtest scores on both tests 
and between TOEFL scores and grades in courses in English as a second 
language. 

Results 



Mean TOEFL scores we e about two-thirds of a standard deviation below 
published norms (cf., Educational Testing Service, 1973). The highest 
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correlations between TOEFL sections were found between Reading Comprehen- 
sion and Vocabulary (.73) and between Engllsli St'-ucture and Writing 
Ability (.67). Factor analysis indicated that all TOEFL section scores 
loaded on a single factor. Scores from the Reading Comprehension and 
Writing Ability subtetts loaded more heavily on this factor than did 
scores fron the other three subtests. Intercorrelations and factor 
loadings are indicated in Table 1. 



Table 1 

TOEFL Subtest Intercorrelations and Factor LoMiogs 







TOEFL 


Subtest 


Loadings on 


TOEFL Subtest 


LC 


ES 


V 


RC 


WA Main Factor 


Listening CoiBprehension (LC) 










.75 


English Structure (ES) 


.57 








.73 


Vocabulary (V) 


.50 


.45 






.70 


Reading Comprehension (RC) 


.66 


.60 


.73 




.89 


Writing Ability (WA) 


.62 


.67 


.52 


.67 


.80 


Total TOEFL 


.82 


.78 


.80 


.89 


.83 



Correlations between CELT and TOEFL subtests ranged from .36 to ,79, 
The correlation between total scores on the two tests was .64, The 
Intercorrelations between CELT and TOEFL subtests are depicted in Table 2, 

Table 2 

Intercorrelations between CELT and TOEFL Subtests 



TOEFL Subtest 



CELT Subtest 


LC 


ES 


V 


RC 


WA 




Structure 


.79 


,58 


.36 


.51 


.63 




Listening Comprehension 


.53 


,52 


.72 


.74 


.65 




Vocabulary 


,59 


.77 


.41 


.63 


.71 




Total 


,52 


.52 


.39 


.43 


.50 





The pairs of subtests that correlated the highest were not those with 
the same labels (e.g. , vocabulary , listening comprehension, and structure). 
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CorrelatlooB between TOEFL scores and class grades were low and generally 
nonsignificant. However, there was a high correlation (.63) between TOEFL 
score and group placeaent. 



Conclusions 



A single factor accounts for nost of the variance on the TOEFL 
subtests. The existence of this factor is also supported by the interre- 
lated nature of tte TOEFL subtests and the moderate to strong correlations 
with CELT subtests. Grades in an intensive English program apparently are 
not predicted by TOEFL score. 
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43. Hwang, K. Y, , & Dlzney, H. F. (1970). Pre<ilctive validity of the 
Test of English as a Foreign Language for Chlneie graduate 
students at an Aaerican university. Educational and Psychologi- 
cal Measurement . 30 , 475-477, 



Purpose 



This paper, based on the first autlu>r*8 doctoral dissertation, was 
concerned with the validity of TOEFL as a predictor of grades in a course 
in English as a second language (ESL) and as a predictor of first term 
grade-point average (GPA) in graduate courses for Chinese students at the 
University of Oregon. 



Method 



Sixty-three s;:.bjects (32 male and 31 female) were chosen for study 
from the 120 Chinese graduate students who enrolled at the University of 
Oregon between fall 1966 and fall 196C. All of the subjects had taken the 
five-part TOEFL prior to admission to the university Ipresianably via the 
International Testing Program] . The subjects h^ received from seven to 
13 years of instruction in English before arriving in the United States. 
The age of the subjects ranged from 22 to 36 years, with a median of 29 
years. The subjects majored in education (ED), social and professional 
services (SPS), natural sciences (NB), social sciences (SS), and architec- 
ture (ARCH). Twenty of the 63 subjects enrolled in an ESL course during 
their first term of graduate study. 



Results and Conclusions 



Table 1 reports means and standard deviations of TOEFL scores and 
CPAs for subjects In various subgroups. 
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Table 1 

Descriptive Statiaticfi for TOEiTL and GPA 



Sex or Major 
Area cf Study 



N 



TOEFL 



SD 



GPA 



MEAN 



SD 



Male 32 504 32.7 3.13 .45 

Female 31 505 42.5 2.97 .51 

ED 21 479 22.3 2.93 .48 

CPS 16 527 38.5 2.88 .47 

NS 11 522 37.5 3.26 .29 

SS 9 513 21.1 2.89 .49 

ABCH 6 489 25.4 2.98 .52 

Entire Group 63 505 38.8 3.03 .37 



Total TOEFL score correlated .66 with ESL course grade for the 20 
subjects vho had taken the ESL course in the first year of graduate work. 
This correlation was significant at the .05 level. Correlations between 
TOEFL score and first-term GPA in various study areas are shown in Table 
2; none of these correlations attained statistical significance. 

Table 2 

Correlations between TOEFL Score and GPA 



Sex or Major 
Area of Study 



N 



Correlation of 
TOEFL with GPA 



Male 

Female 

ED 

SPS 

MS 

8S 

ARCH 

Entire Group 



32 
31 
21 
16 
11 
9 
6 
63 



.17 
.18 
.21 
.05 
.22 
-.32 
.69 
.19 



ERIC 



The results of the study indicated that TOEFL was a relatively good 
predictor of ESL course grades for Chinese graduate students at the 
University of Oregon but not a very good predictor of graduate course 
grades. Further studies involving Chinese students in other universities 
•nd colleger, are needed, as well as studies focusing on other groups of 
foreign students. 128 
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44, Irvine, P., Atai, P., & Oiler, J. W., Jr. (1974). Cloze, dictation 
and the Test of English as a Foreign Language. Language Lea rn- 
ing , 24, 245-252. 



Purpose 

This study investigated the perfomance of foreign students on a 
cloze test, a dictation test, and the five-part TOEFL. One objective was 
to detenaine whether TOEFL scores would correlate with the other oeasures 
in a way consistent with integrative proficiency testing tl^ory. A second 
objective was to verify that perfonaancc on the cloze test and dictation 
test, both presented in English, w>uld correlate with each other for a 
group of exaalnees based in a foreign country. 



Method 

The subjects were 159 native Farsi speakers located in Tehran, Iran. 
[No other details are given on the characteristics of the subjects.] The 
five-part TOEFL was administered by trained staff according to standard 
procedures. After the test, volimtary participation to take the cloze and 
dictation tests the next day was requested. 

The dictation test employed two separate passages. The easier 
passage was taken from a college-level text on community health; the more 
difficult passage came from Scientific American . The passages were read 
by a native English speaker and required about 20 minutes to read. The 
total dl'-tatlon score was the total number of words written by the subject 
in orlglt.dl sequence for the two passages combing. The cloze test was 
based on a 394-word passage taken from an English text. Every seventh 
word in the passage was deleted, except for the first 43 words, which 
remained Intact. Two zloze test scores were derived. The exact cloze 
score was the total nimber of verbatim reproductions of the deleted words. 
The acceptable cloze score wss the total number of words that were accept- 
able substitutes, as judged by a native speaker of English. 

Results 

The exact cloze score correlated .94 with the acceptable cloze score, 
and the two cXozc scores correlated in a very similar fashion with other 
measures. The acceptable cloze score correlated .75 with the total 
dictation score, a correlation that was higher than the correlation of acy 
TOEFL section with any other TOEFL section. The acceptable and exact 
cloze test scores correlated more highly with the TOEFL Listening Compre- 
hension subscore than with any other TOEFL subscor«>. 
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Correlatlons were computed between each Individual TOEFL section 
and the coobinatlon of all roaaining sections; the close and dictation 
test were also correlated with these conbined scores. With one exception, 
cloze test scores proved to be better predictors of the coobinatlor of 
TOEFL section scores, excluding a given section, than did the score on the 
excluded section or the dictation Cast score* Correlations of cloze 
scores with the various combinations of four (or all five) TOEFL sections 
ranged frotn .74 to .81. 



Conclusions 



The pattern of correlations obtained is Judged to support the pre- 
vious claim that dictation and cloze tests represent integrative language 
proficiency skills. For exanple, there was a moderately high correlation 
of .69 between the dictation score and the TOEFL Listening Comprehension 
score. This finding bolsters the contention that separate tests, empha- 
sizing different language tasks, measure the same underlying range of 
integrated language skills. 

It Is concluded that the TOEFL total score provides little Interpret- 
able Information beyond that provided by the cloze test, the dictation 
teat, and the TOEFL Listening Comprehension subtest. It is also concluded 
that the patterns of Intercorrelations among scores do not support the 
notion that the separate TOEFL subscores measure highly differentiated 
language skills. 
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45. Jameson, S. C, & Malcoln, D. J. (1973). TOEFL — The developing 
years. International Educational and Cultural Exchange . 8, 
57-62. 



Purpose 



This descriptive paper provides an account of the events in the early 
history of TOEFL. [See also Falser, 1965; Oiler & Spolsky, 1979. Note 
that this suoaary, as mil as swaaries of other papers, directly reflects 
statements made by the authors. Thus, the aumsary la written in the 
present tense and refera to groups and organisations hy their titles at 
the time of writing, even though soae of those oi^anisations have since 
changed their names.] 



Discussion 



At a 1961 conference on testing, the need was expressed for a ayateor' 
atic method of assessing the English proficiency of foreign applicants to 
U.S. colleges and universities. As a result, the National Council^n the 
Testing of English as a Foreign Language was established, ii4l>(l^ns 
representatives of approxiaately 30 organisations. With support from the 
Ford and Danforth Foundations, the Council began the groundwork toward 
development of the Teat of English as a Foreign Language, and actual test 
development was well underway by the summer of 1963. The test, which 
consisted of five parts, was first administer^ in February 1964 to 920 
examinees in 34 countries. 

The council soon recognized that a well-established organization 
would be needed to sponsor the TOEFL program. In July 1965, the College 
Entrance Examination Board (CSEB) and Educational Testing Service (ETS) 
assumed cooperative responsibility for the prograa. CEEB assumed respon- 
sibility for promoting use of the program by collies and universities and 
to ensure that the program would be responsive to the needs of those 
institutions and of the exaainees. ETS assumed responsibility for opera- 
tion of the program, including preparation of teat forms, examinee regis- 
tration, establishownt of test centers, test scoring and score reporting, 
statistical analysis, research, and other functions. 

The TOEFL program's growth is seen in marked increases from the year 
1964-65 to the year 1971-72 in number of exrainees tested (from approxi- 
mately 2,400 to 64,000), average number of test centers per administration 
(from 80 to 400), and other statistics. 

The large majority of colleges and universities in the United States 
now require, or strongly recommend, that foreign applicants take TOEFL. 
Also, the test is used by many government agencies, busineases, and 
foundations, and it is required by many organisations as a basis for 



131 



126 



ERIC 



licensure or accreditation of professionals educated outside the United 
States. TOEFL 's advantages include the security of the test, the value of 
the test scores for adalsslon and placeaent, and the fact that the test is 
offered in so nany locations abroad. 

In 1966, CEEB and ETS appointed a six-aeaber Exaalner Coaaittee, 
comprised of specialists in linguistics, psycholinguistica, and the 
teaching of English as a foreign language. The Exapiner Coaiattee reviews 
new test fonts, provides advice regarding research, and brings to bear 
knowledge of new developaents in the field. The National Advisory Council 
on TOEFL, which evolved froa the original TOEFL Council, serves to advise 
CEEB and ETS on matters of general policy regarding the teat. The Council 
consists of representatives of undergraduate institutions, graduate and 
professionsl schools, teachers of English as a foreign language, adais- 
sions officers, and various agencies. 

Several procedural changes have been made over the years. Scores are 
now given directly to exaainees as well as to Institutions; the adminis- 
tration of test centers and of the registration process have been decen- 
tralized in many places; four International administrations rather than 
three are now offered each year; and a totally new test form is used at 
each International administration. Also, an Institutional Testing Prograa 
has been established, through which TOEFL forms are provided to institu- 
tions for administration to their own students. Further, walk-in test 
centers were established, whereby students could take TOEFL at times other 
than the International administrations test dates. 

The program has benefited from cooperation between governaent 
agencies and Institutions of higher education. From the outset, support 
has come from the Bureau of Educational and Cultural Affairs of the 
Department of State, and the Agency for International Development, 
Encouragement has also come frwa the Institute of International Education, 
the American Friends of the Middle East, and the Latin American Scholar- 
ship Progrms of the African University. The widespread use of TOEFL has 
also been due, In part, to strong recommendations by the American Associa- 
tion of Collegiate Registrars and Admissions Officers and the National 
Association for Foreign Student Affairs that U.S. institutions require 
foreign applicants to take TOEFL. 

With the growth of the TOEFL program, vatic .t , rjUcations relating 
to the test have been prepared, including interpretive iB*»-rf . s for TOEFL 
users as well as a guide for students. A technical manual is planned that 
will summarize research on TOEFL and present detailed Information about 
its psychometric characteristics. 

As TOEFL continues to be used, it is important that scores •>€ 
properly interpreted by those who use them. TOEFL should not be regarded 
as an aptitude test or as a predictor of academic success, and cutoff 
scores should not be used. Research should be conducted by institutions 
that use the test so they can establish norms by which to make proper 
decisions at the local level. 
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46. Johnson, D, C. (1977). The TOEFL and domestic students: Conclusive- 
ly inappropriste. TESOL Quarterly , 11 , 79-86. 



Purpose 

This study examined the perforaance of na*:ive English speakers on 
TOEFL, partly to deternlne the appropriateness of cutoff scores on the 
test used by the University of Tennessee for admission of foreign stu- 
dents. 



Method 



The subjects MBre 173 American freshmen and sophomores enrolled in a 
psychology course at the University of Tennessee. Of the 173 subjects, 
152 were from Tennessee, 96 from urban areas, and 56 from rural areas 
(towns with populations of less than 30,000). Scores on the American 
College Test (ACT) indicated that this sample was representative of the 
student body at the university. [The ACT is described briefly in Summary 
No. 5, Andalib, 1976.] 

Participation in the study vas voluntary and resulted in research- 
participation credit and an opportunity to win prize money. The five-part 
TOEFL was administered to the subjects as part of the study* 



Results and Conclusions 



Mean scores on the five parts of the test were Listening Comprehen- 
sion, 69.9 (of a possible 73); English Structure, 64*3 (of 66); Vocabu- 
lary, 65.2 (of 69); Reading Comprehension, 56.6 (of 66); and Writing 
Ability, 58sO (of 65), for a total score of 628. Thus, consistent with a 
similar study conducted at a western state university (Ang'^ff & Sharon, 
1971), the scores were relatively high, and the lowest subscores obtained 
were those for Reading Comprehension ami Writing Ability, with the levels 
of performance on the other subtests also generally similar to those of 
the earlier study. The relatively high scores of this native-English- 
speaking sample suggests that the minimal TOEFL score of 475 established 
for admission to the University of Tennessee is not unreasonably strin- 
gent. 

Subjects from urban and rural areas were compared with respect to 
their Listening Comprehension scores. The difference between them was 
slight and nonsignificant, indicating that speaking of regional dialects 
has little relationship to ability to perform this listening task, perhaps 
because of the subjects' exposure to radio and television. 



133 



128 



The correlation between total TOEFL score and ACT English score vas 
.67 (compared with .6A in the Angoff & Sharon study). That this correla- 
tion was not higher is attributed to the difference in difficulty of the 
two tests and the fact that they ar6 intended for use with different 
populations. 

Scores for this sample were compared with those available for the 
2 15,486 foreign candidates who took TOEFL worldwide between October 
1966 and June 1971. The mean of the present sample, 628, Is considerably 
higher than the mean of 490 for the foreign candidates. Also, the stand- 
ard deviation of this sample, 35,1, is considerably lower than the 80.0 
observed for the foreign candidates. 

In general, the results demonstrate that TOEFL is inappropriate 
for use with domestic students. The relatively high mean score of the 
present sample and the narrow distribution of scores reflect the fact that 
the test was not developed to discriminate among native speakers of 
English. 
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47. Konvlchayungyuen, N. (1978). Test of English as a Foreign Language 
as a predictor of actual English proficiency (Doctoral disser- 
tation, Florida State University, 1977). Dissertation Abstracts 
International , 38 , 6666A. (University Microfilms No. 7804978) 

Purpose 

Previous validity studies have usually assessed TOEFL's relation to 
other testa of English proficiency or have examined TOEFL's ability to 
predict academic success. This study, in contrast, examined the relation 
between TOEFL scores and teachers' ratings of students' English profi- 
ciency. 



Method 

The subjects were 57 students who had taken TOEFL and were selected 
from 108 foreign graduate students enrolled in 15 different progrras in 
the College of Education at Florida State University between 1974 and 
1977. The sample consisted of 32 males and 25 females; 40 were Ph.D. 
students. T'aelr ages ranged from 22 to 46, with a mean of 30.8 years. 
They were drawn from 22 countries, representing the following regions: 
Far East Asia (N - 8) , Southeast Asia (N - 14), Middle East Asia (_N - 15), 
South America - 11), and others ( N - 9). 

The five-part TOEFL had been administered to the subjects prior to 
admission to the university [presumably via International or Special 
Center administrations). 

Each subject's English proficiency was rated in four areas: writing, 
reading and vocabulary, aural comprehension, and speaking. The rating 
scale had five levels: elementary proficiency. Intermediate proficiency, 
minimal academic proficiency, partial academic proficiency, and full 
academic proficiency. Each subject was rated by one or more faculty 
members; only faculty members who knew a subject well provided ratings for 
that subject. A subject's ratings were averaged across judges and sumn^d 
over all four areas to yield a total rating of English proficiency. 



Results and Conclusions 



Re 1 labilities of the ratings were determined by a special procedure 
described by Ebel^ since different subjects were rated by different 



^Ebel, R. L. (1951). Estimation of the reliability of ratings. 
Fsychometrika, 16, 407-424. 
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numbere of faculty iieiber*. The obtained reliabilities, which were based 
on three ratings for each subject, were writing, .29; reading and vocabu- 
lary, .32; aural conprehension, .39; speaking, .28; and total of the 
ratings, .71. The reader is referred to overall data reported in Educa- 
tional Testing Service (1973) for the reliabilities of the TOEFL subtests. 

The subjects in this sample averaged 520.75 for total TOEFL score, 
which is more than one-half of a standard deviation above the mean of all 
foreign candidates in the field of education. Mean part scores on the 
TOEFL Listening Comprehension, English Structure, Vocabulary, Heading 
Comprehension and Writing Ability sections, respectively, were 47.43, 
50.80, 55.09, 55.39, aiu! 51.02. 

Mean English proficiency ratings for writing, reading and vocabulary, 
aural comprehension, and speaking, respectively, were 3.98, 4.33, 4.04, 
and 3.80, for a mean total score of 16.16. 

Visual inspection of the data shows that the highest TOEFL scores and 
ratings were earned by subjects from Southeast Asia; women earned higher 
TOEFL scores and ratings than did men; and master's students earned higher 
TOEFL scores but lower ratings than did doctoral students. 

High intercorrelations among ratings in the four different areas of 
proficiency were observed (.84 to .91), probably due to a halo effect in 
the ratings — that is, the tendency for a rater to be influenced by percep- 
tion of a subject's skill in one area in Judging his or her skill in 
another. 

The correlation between total TOEFL score and total rating was .53, a 
moderate but highly significant relationship. Correlations between total 
TOEFL score and rating for ^ach of the four areas of rated proficiency did 
not differ greatly, ranging from .45 to .55. It appears that TOEFL is a 
moderately good predictor of English proficiency as determined by the 
rating procedures used in this study. Separate correlations were computed 
between TOEFL and proficiency ratings for each sex and planned degree, and 
no significant differences in correlations were tound as a function of 
these variables. Thus, TOEFL's ability to predict rated English profi- 
ciency does not appear to depend on the sex or planned degree of the 
subject. 

Tables are presented Indicating the kinds of rated proficiency levels 
associated with each of several ranges of TOEFL scores. It is concluded 
that more data need to be gathered for subjects with TOEFL scores below 
500 in order to derive a more reliable content-referenced scale. 
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48. Ku, E. J., & FrlBbie, D. A. (1979, April). An exaalnatlon of the 
confounding in aeasureB of foreign language listening conprchen - 
sion . Paper presented at the meeting of the National Council or 
Measur&aent in Education, San Francisco. 



Purpose 



The moderate to high relationship between measures of reading coapre' 
hension and listening comprehension on TOEFL and sliBllar tests nay be due 
to the structure of the typical test of listening coaprehension, as it 
requires the subject to read the response alternatives. To address this 
issue, this study examined the correlation between TOEFL Reading Compre- 
hension and each of twi measures of listening comprehension: (a) the 
TOEFL Listening Comprehension subtest, and (b) a completely oral modifica- 
tion of that subtest. 



Method 



The subjects were 150 foreign students from Hong Kong and Taiwan, 
undergraduate and graduate student volunteers at the University of Illi- 
nois at Urbana-Champaign. Of these 150 subjects, 30 comprised the sample 
for a pilot study, and 120 formed the main sample^ In the pilot study, 
all 30 subjects were administered the Reading Comprehension subtest; 15 
were also administered the traditional Listening Comprehension subtest and 
15 the experimental listening test. In the main study, 120 subjects were 
administered the traditional Reading Comprehension subtest; 60 were also 
administered the traditional Listening Comprehenr Ion subtest and 60 the 
experimental listening test (with presentation ci ?r counterbalanced to 
eliminate possible order effects). 

The traditional Listening Comprehension and Reading Comprehension 
subtests of the five-part TOEFL were administered as described in the 
Introduction to the summaries. Fifty items were tised in the Listening 
Comprehension test. The experimental (oral) listening test used a 
true-false format and included 64 items (to ensure that testing time would 
be equal to that for the traditional Listening Comprehension subtest). 
For example, in one part of the test the subject heard '*Wbcn did Tom come 
home?" "By taxi"; the subject was then asked to indicate whether the 
response heard made sense in relation to the qiiestion. The measures used 
in the pilot study and in the main sttidy were basically the same except 
that eight of the 64 items in the experimental listening erasure were 
revised in preparation for the main study to Improve the test's discrimi- 
nation power. 
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Results 



In the pilot study, KR-21 rixiabillty of the experimental listening 
test was found to be lower (.65) than that of the traditional Listening 
Comprehension subtest (.81). Attenuated correlations between Reading 
Comprehension and listening scores were .90 for the traditional Listening 
Comprehension subtest and .47 for the experimental listening test. 

In the main study, KR-20 reliability of the experimental listening 
test was .73 and of the traditional Listening Comprehension subtest, .86. 
In contrast with results for the pilot sample, the data showed nearly 
Identical correlations between the Reading Comprehension and the tradi- 
tional Listening Comprehension subtests (.59) and between the Reading 
Comprehension subtest and the experimental listening test (.57); correla- 
tions corrected for attenuation were .78 and .77. 



Conclusions 



Despite a suggestion to the contrary from the pilot data, the TOEFL 
Reading Comprehension subtest did not correlate more highly with the 
traditional Listening Comprehension subtest than with the experimental 
listening test. Thfe rationale for expecting a higher correlation In the 
first case than the' second still seems logical, however, since the tradi- 
tional TOEFL Listening Comprehension subtest requires the reading of 
response alternatives. Perhaps both the reading and traditional listening 
tests used here were too easy for these subjects (high means and low 
standard deviations of scores suggests that this may have been the case) 
and that a more diverse sample might well show the expected difference in 
correlations, ^^lso, the quality and reliability of the experimental 
listening measure could be further improved. In general, conr.lusions 
should not be dr wn until the study can be repeated on a larger scale, 
with a more diverse group of subjects and a revised experimental list-^ning 
test. 
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49. Little, B. G, (1980). Acquisition of deteminer functions . Unpub- 
lished master's thesis. University of California~Los Angeles. 



Purpose 



This study exatnined Japanese students' knowledge of when to use 
definite, indefinite, and zero articles in Englioh, a particular problem 
for Japanese students because their language lacks a comparable article 
system. Questions under investigation were: (a) Are there clear differ- 
ences in article usage between high and low proficiency students? (b) Do 
measures of article usage correlate with scores on TOEFa. and the Michigan 
Placenent Test? and (c) Are there patterns showing tte relative difficulty 
of different functions served by the different articles? This summary 
focuses primarily on the relation to' TOEFL and the Michigan test; the 
various functional patterns are treated only briefly. 



Method 



The subjects for the study were 27 Japanese students, 10 female and 
17 male, in the Intensive English program at the University of California 
at Riverside. Two proficiency groups were defined according to scores on 
the Michigan Placement Test (high group: 64 to 95; low group: 28 to 54). 
The 27 subjects were those (among 94 total Japanese students in the 
program) who (a) were present for the duration of the study, (b) were in a 
writing class, (c) completed both parts of the Article Survey described 
beJrrf, and (d) had Michigan test scores In the ranges Indicated above. 

Two measures of article usage were used: (a) the Survpy of the 
English Determiner System, or "Article Survey" developed by the author, 
and (b) article usage in a written composition. The Article Survey was a 
two-part test. The first part consisted of 50 multiple-choice items. For 
each Item, a sentence containing a blank space was presented and the 
subject was to indicate which of the followltig choices correctly filled 
the space: "a," "an," "the,** "no determiner," or "optional." The second 
part was a modified cloze test, with 35 blanks to be completed using the 
correct article (or zero article). The text was edited from a story In a 
fourth-grade science book. In earlier research with 40 nonnative adult 
ESL students, KR-20 reliability of the Article Survey was found to be .89; 
the correlation between parts was .75. Although a second study of 30 
advanced English students given Part I showed a reliability of only .49, 
this was probably due to restriction In range. Part I reliability for the 
two samples combined was .78. 

In the present study the composition was written during a regularly 
scheduled writing class, with 50 minutes allowed for its completion. The 
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topic for the coBpositloii was "A t3^1caX African cwboy," and 30 relevant 
vocabulary words nere provided. Article usage was analyzed in the follow- 
ing manner. The presence or absence of the reqrlred article was deter- 
mined for each obligatory context. The nimber of errors was subtracted 
from the number of obligatory contexts, and the difference was divided by 
the number of obligatory contexts to yield the subject's score. 

Scores were available on TOEFL [presumably the three-part version, 
administered via the International or Special Center program]. The 
Michigan Placement Test was administered to all subjects at the outset of 
the academic quarter, a few weeks before the beginning of this study. 
{The structure of this test is not described.] 



Results and Conclusions 



Scores on the Article Survey were significantly higher for the high 
than the low proficiency group. Separate analyses for each of the two 
articles, "a" and "the," and for the zero article (i.e., the absence of an 
article) also showed tl^ two groups to differ significantly in all three 
cases. Analyses according to specific f miction of each article were also 
performed (four functions of the indefinite article, five of the definite 
article, and two of the zero article; e.g., functions of the Indefinite 
article include (a) the number one, (b) one undifferentiated specimen In a 
class, etc.). In these analyses the group differences were generally too 
small to be sigcHlf leant . For the written composition, the trend was in 
the direction of a difference between proficiency groups in correct 
article usage, but the group difference in this case was not significant 
(£ - .10). 

The Article Survey correlated .71 with the Michigan Placement Test 
and .58 with TOEFL (both correlations were significant). Thus, the 
Michigan test appears to be a moderately good predictor of overall per- 
formance on the test of article usage, and TOEFL also a moderately good, 
though a slightly less valid, predictor. Separate scotes for the various 
article functions did not correlate well with TOEFL and the Michigan test, 
as the only significant correlations were those involving the general use 
of the zero article (TOEFL: _r - .65; Michigan: jr - .74). 

Article usage scores derived from the composition correlated .47 with 
TOEFL and .37 with the Michigan test. While both of these correlations 
were significant, apparently the scores on these t ^sts do not provide very 
accurate prediction of correct article usage in writinflr Overall the ^ 
results involving correlation with TOEFL and the Michigan test indicate 
that scores on these tests do not strongly reflect the subject's ability 
to use the English article system correctly. Scores on these tests do 
not, therefore, bear directly on development of strategies for teaching 
article functions. 
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Other results of interest include the folXoiring: (a) Various func- 
tions of the articles shovied roughly thB soae rank ordering of difficulty 
for both the low and high proficiency groups, (b) Uhen tlw data for the 
Article Survey and th« coi^sition ware combined, article ooission consti- 
tuted 31 percent of all errors; incorrect ufe of the Indefinite article, 
25 percent of errors; and incorrect use of the definite article, 24 
percent of errors, (c) In the eoaposltions, more than half of the low 
proficiency group failed to produce situations involving seven of the 11 
functions, snd more than half of the high proficiency group failed to 
produce situations Involving three of the sane seven functions. This last 
result supports earlier daiaui that second language learners tend to avoid 
certain situations in which they are unsure of correct article usage. 

Soth the low and high proficiency groups had the greatest difficulty 
with the definite article followed by the indefinite article, then th& 
zero article. This observed order of difficulty differs from that found 
in two other studies cited— one involving Japanese subjects only, the 
other involving 29 different language groups— in which the order of 
difficulty froa highest to lowest was indefinite, definite, and zero 
article* Reasons for the difference among studies remain to be deter- 
mined. Perhaps the difference is due to the different degrees of profi- 
ciency of the subjects in the various studies or differences in the 
instruments used. 
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50* Martin, G. M. (1971). A model for the cultural and statistical 
analysis of academic achieveiaent of foreign graduate students at 
the University of North Carolina at Chapel Hill (Doctoral 
dissertation* University of North Carolina at Chapel Hill, 
1971). Dlsae^tatlon Abstracts International, 32, 2311A. 
( Unlver el ty mcrofilas No. 71-30, 578) 



Purpose 



This study addressed four sets of questions regarding academic 
performance of foreign graduate students at the University of North 
Carolina: (a) Do students who have taken TOEFL and been admltt^ with 
satisfactory scores differ from those who have been admitted without 
having taken TOEFL? (b) Does tte difference between the above-mentioned 
two groups vary as a function of the students' geographic origin or major 
field of study? (c) Are there differences in academic performance among 
students from different geographic r^lons or among students enrolled in 
different major fields? (d) For those who have taken TOEFL, what is the 
relation between academic success and scores on each of the TOEFL sub- 
tests? 



Background 



For the past five years (1964-1969), the University of North Carolina 
at Chapel Hill has begun to require TOEFL of all foreign applicants. 
However, the requirement has not been strictly enforced, and only about 25 
percent of enrolled students have taken TOEFL and received the minimum 
score required by the imiversity (see below). For others, English profi- 
ciency has been determined on the basis of (a) certification by a U.S. 
government official or local English professor, (b) interview with a 
university representative, (c) school record, or (d) the student's self- 
evaluation or correspondence in English. 



Method 



The subjects were 144 male foreign graduate stttdents ^o had begun 
their studiek at the Universl*-y of North Carolina at Chapel Hill between 
fall 1964 and l|all 1969. (Too few females were available for inclusion in 
the sample.) ^ total of 72 subjects had taken the five-part TOEFL and 
received the required minimum score. (At the time of the study, the 
university's re4uired minimum was a total score of 500 and a score of 50 
on each subtest; however, previous scores had not met these criteria in 
every case.) These subjects were matched, with respect to geographic 
origin and major field, with 72 other students who had not taken TOEFL. 
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Forty couatri«s were represented. For seatiftic*! anelyelt* tubjects were 
grouped according to three sajor geographic r^lons: Europe and South 
Anerlca (including Cyprus, the Ztoainican Eepid>lic, and Puerto Rico; N - 
44); Middle East (N • 42); and Asia (N - 58). A total of 35 major fields 
were represented; for analysis, these have been subdivided into four 
groups: the huaanities (N " 22); the social sciences (including philos- 
ophy and political scienc'e) ( JK * 40); the sciences (natural sciences and 
aatheaatics) (N » 32); and professional schools (N » 48). [Major fields 
for ttio students" are not reported.] " 

Data from the five-part TOEFL were available from the subjects* 
records [presumably based on administrations of the test in the Interna** 
tional Testing Progrsml. 

Grade-point averages (CPAs) were calculated by the investigator. 
Although the University of North Carolina has no grade-point system as 
such, the following grades are given in each course: H (highest), P, L, F 
and Incomplete. The investigator assigned ttm following nuoibers to each 
grade: H-3, P-2, t">l, F"0, and Incomplete - 0. An average was 
then calculated, iidth the grade for each course weighted by the number of 
credit hours for that course. 



Results and Conclusions 



^ tests were used to test for differences among various subgroups. 
It is'noted that, because of the large number of ^ tests computed, some 
may have been significant by chance. 

The first analyses of Interest were comparisons of students who had 
taken TOEFL with those who had not. T tests showed the TOEFL takers to 
have significantly higher CPAs for both the first s^ester (2.20 vs 1.85) 
and the second semester (2.13 vs 1.67). A breakdown by geographic area 
shows the difference between TOEFL takers and non-TOEFL takers to be 
significant for all three major groups: Europeans and South Americans, 
Asians, and Middle Eastern subjects. A breakdcwn by major field shows 
that TOEFL takers had significantly higher f Irst-seaester grades than 
non-TOEFL takers for the sciences and the professions and significantly 
higher second-semester grades for the sciences, professions, and social 
sciences. 

Analyses were also performed comparing the grades of the major 
geographic groups, separately for TOEFL takers and non-TOEFL takers. Only 
two of 12 possible differences were significant, both involving first- 
semester grades: (a) Middle Eastern subjects outperformed Asians £anong 
those not taking TOEFL, and (b) Middle Eastern subjects outperformed 
Europeans and South AiKrlcans among those taking TOEFL. 

Comparisons among major-field groups showed eight of 24 differences 
to be significant. For TOEFL takers, first-semester grades mxe higher 
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for Che sciences than for (a) the huaanltles, (b) the social sciences, 
and (c) the professions; second-senester grades were lower for thB humani- 
ties than for (a) the social sciences, (b) the sciences, and (c) the 
professions. For non-TOEFL takers, second-semester grades for the sciences 
exceeded those for the social sciences and for the professions. 

Correlations were coaputed between CPA and each of the TOEFL sub- 
scores for the 72 subjects who had TOEFL scores; correlations of CPAs 
(first- and second-SMieeter) with TOEFL subscores ranged from -.17 to -.09 
(all nonsignificant). 

In general, subjects who took TOEFL and received scores satisfactory 
for adalsalon perfrraed better In both semesters than did those who had 
not taken TOEFL, and this was true for subjects from all three geographic 
groups. Thus, TOEFL appears to be a factor In the academic achievement of 
foreign male graduate students, at least those In the present sample. 

For those admitted on the basis of satisfactory TOEFL scores, the 
lack of a significant correlation with CPA for any of the subtests indi- 
cates that, for these students, a relationship between TOEFL and academic 
performance cannot be inferred and no one TOEFL subtest is a better 
predictor than another. 
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51. Maxwell, A. (1966). A coapariBon of TOEFL and the UCB/EFL teit . 

Unpublished aaater'i tneeis, Sacreaento State College, CA. 



Purpose 



Thia study Inveatlgated the validity of the University of California, 
Berkeley, Test of English as a Foreign Language (UCB/EFL) by correlating 
it with TOEFL. The study also examined the predictive validity of both 
measures as reflected by correlations with students* overall grade-point 
average (CPA). 



Method 



The five-part TOEFL and tte UCB/EFL were administered to 238 foreign 
students enrolled at the University of California, Berkeley, during the 
fall of 1964. The UCB/EFL consists of five parts. Part 1 consists of 55 
multiple-choice items covering grsaoatical points included in a book on 
English sentence patterns. Part 2 consists of 20 multiple-choice items 
based on everyday vocabulary. Part 3 contains 10 multiple-choice items 
testing punctuation points for which there are firm rules. Fart 4 consists 
of 20 low-frequency words that tmist be spelled by the subject. This part 
emphasizes major spelling patterns and the ability to discriminate English 
sounds ("r" vs. "1", etc.). Part 5 is a 100-150 word dictation test. 

Student CPA for the 1964-65 academic year was based on content 
subject areas only* Thus, grades in physical education and remedial 
speech were deleted frma the data. Separate correlations were calculated 
for separate subject areas. 



Results 



The correlation between TOEFL and the UCB/EFL was .87. The subjects 
showed a mean TOEFL score of 525 with a standard deviation of 79. Corre- 
lations with CPA for groups larger than 20 are depicted in Table 1. 
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Tabic 1 

Correlations of TOEFL and DCB/EFL with CPA 
for Various Groups of Subjects 



Correlation Significance 
with GPA Levels 
Subject Group N TOEFL UCB/EFL TOEFL IffiB/EFL 



All subjects 238 

Undergraduates 191 

Graduate students A 7 

Econoaics/buslness majors 39 

Engineering msjors 101 

Languagj^^^terature aajors 21 

Natural science najors 27 
Subjects froa Europe ^ 105 
Subjects froB the Middle East 27 

Subjects from the Far East 61 

Males 202 

Females 36 



.17 


.11 


.01 


ns 


.58 


.53 


.01 


M 


.02 


-.01 


ns 


ns 


.45 


.37 


.01 


.05 


.24 


.20 


.05 


.05 


.16 


.07 


ns 


ns 


.38 


.16 


.ISS 


ns 


.07 


.01 


ns 


ns 


.50 


.49 


.01 


.01 


.24 


.15 


ns 


ns 


.15 


.12 


.05 


ns 


.33 


.15 


.05 


ns 



Conclusions 



The high TOEFL-UCB/EFL correlation indicates that both tests are 
measuring similar characteristics. Both are valid predictors of GPA at 
the undergraduate *level, while neither is s valid predictor at the gradu- 
ate level. Considering the ninber of cases in which each test was signi- 
ficant, it would appear that TOEFL is the better predictive measure 
overall. 
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52. Mullen, K, A, (1978) » Determining thB effect of uncontrolled sources 
of error in a direct test of oral proficiency and the capability 
of the procedure to detect iaproveaent following classroom 
instruction. In J. L, D, Clark (Ed*), Direct testing of speak- 
ing proficiency; Theory and application , Princeton, NJ: 
Educational Testing Service. 



Purpose 



This study sought to determine the reliability of an oral Interview 
procedure and to determine the effects of classroom instruction in English 
as a second language (ESL) on the amount of improvement in both oral 
Interview and TOEFL scores and on correlations between TOEFL and Interview 
scores. 



Method 



The main sample consisted of 107 students in the ESL program at the 
University of Iowa who were given an oral Interview both before and after 
a period of Instruction, For 15 of these subjects, the five-part TOEFL 
Was administ 'red both before and after instruction; three additional 
subjects were given the Listening Comprehension subtest of the TOEFL at 
both times. 

Each oral interview was conducted by two experienced ESL instructors 
and lasted 15 t 20 minutes. One interviewer took the lead in the inter- 
view, while the ^ther listened and occasionally interjected questions to 
facilitate the conversation. After a general discussion to put the 
subject at ease, the subject was asked about a topic on which he or she 
could speak with authority for some time — e,g,, regarding his or h(0t 
family, education, academic interests, goals, opinions, impressions, or 
attitudes. After each Interview the two interviewers rated the subject on 
five scales: listening comprehension, prontmciation, fluency, grammar, 
and overall proficiency. For each scale, five rating categories were 
used: poor, fair, good, above average, and excellent; interviewers were 
allowed to give ratings midway betv^en any two categories, thus producing 
a nine-point rating system. 

The subjects were interviewed twice, once when first evaluated for 
placement, and again after a semester of instruction. Instructors Inter*- 
viewed students who had not been in their classes, and some new students 
were interviewed during the second session to ensure that interviewers 
could not distinguish old and new students. 
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Results end Conci«uiioa« 



Interview reliabilltlet obtaln»3 for the first and aecootd aeaalons 
were not significantly different froa each other for any of the five 
ecales, and the mean rellabllitlaa, for theae five acalea ranged froa .78 
to .89. (Note that oean ratlnga provide! by the tw intervlewera for each 
subject did not differ algnlf Icantly, and, la rll aoalyaea, a atibject*a 
Interview score was the average of the two jt^cervlewera- ratings. These 
rellabllltlea were based on US subjects Intervleued In the first session 
and 152 subjects In the second. } 

T tests for the aaln saapXe of 107 subjects sbmied that there was 
significant laproveaent from the first (pre-lnstructlon) session to the 
second (post-Instruction) aesslon on all five Interview scales. For the 
subsaaple of 15 subjects for whoa TOEFL scores were available (18 aubjects 
for Listening Coaprehenslon) , significant iaproveoient vaa evident on four 
of the five Interview scales (all but graaaar) as liell as the overall 
score and cn three of the five TOEFL subtests (Listening Coi^prehension, 
English Structure, and Writing Ability) as well as the total score. 

Of the TOEFL subtests, Listening Conprehenslon is most relevant to 
the topic of this paper, assessaent of oral proficiency. Since this 
subtest requires some reading as well as listening, it is possible that 
laproveaent on this subtest could be at least partly due to laproveaent in 
reading proficiency. However, since the oean score on the Reading Coapre- 
henslon subtest did not also show a significant laproveaent froa the first 
to the second testing session, the performance change in Listening Coapre- 
henslon appears to be due to a change in actual aural proficiency. This 
Is consistent with the observed laproveaent in scores on the listening 
comprehension scale of the Interview, as well as with the laproveaent in 
the other oral skills shown in the interview: pronunciation and fluency. 

The TOEFL English Structure subtest might be regarded as a measure of 
passive control over English, since this test allows conteaplatlon of 
possible choices, whereas the Interview might be regarded as a s^asure of 
active control. If this is true, and if passive control is greater than 
active control it would be expected that TOEFL and Interview scores uould 
not be highly correlated in the first testing period; however, ^as the 
subjects receive Instructiun and experience in English, they should 
develop active control as well as passive control, and thus the correla- 
tion between the interview and TOEFL scores should be higher for the 
second testing period. This was Indeed the case. None of the correla- 
tions between TOEFL subtests and the interview was significant in the 
first session (£*s ranged up to .37), whereas- several correlations i«re 
significant In The second session, particularly for two TOEFL subtests: 
TOEFL Listening Comprehension correlated significantly with interview 
scores in listening (.45), grammar (.43), and overall score (.54); and 
TOEFL Vocabulary correlated significantly with interview scores in listen- 
ing (.46), pronunciation (.70), grammar (.46), and overall score (.44), 
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In general, ehe result* denonatrate a reasonably reliable Interview 
procedure for teating aubjecta* apeaklng proficiency. Low correlations 
between TOEFI* and interview scores in the first session suggest that, 
initially, proficiency is low and passive control exceeds active control. 
However, the increase in correlation between TOEFL and interview scores 
as the student receives language instruction suggest a dialnution in the 
difference between active and passive control of English. 
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53. Mullen, K. A. (1979). An alternative to the cloze test. In C. A. 

Yorlo. K. Perklna, & J. Schacter (Eds.), On TESOL *79, The 
learner In focui (pp. 187-192). Washington, DC: Teachers of 
English to Speakers of Other Languages. 

Purpose 



This study coapared performance on an editing test with performance 
on a cloze test and two direct tests of English proficiency. The two 
direct tests were an oral interview and a writing task. Relationships of 
these measures with TOEFL were also investigated for subjects with TOEFL 
test scores. 



Method 



The subjects were 54 foreign candidates for admission to the Univer- 
sity of Louisville, including some who were seeking to transfer from 
community colleges. The subjects, who varied in their exposure to English 
language programs, were administered four proficiency tests: an editing 
test, a cloze test, a writing task, and an oral interview. Scores on 
the three-part TOEFL were available for 22 of the subjects Ipresinably 
obtained in International or Special Center administrations]. 

The editing test, based on a passage written at a seventh-grade 
reading level, required that the subjects cross out 50 words that did not 
belong in the passage. The 50 words in question had been taken from the 
original passage and had been subsequently inserted at random into that 
passage. Two performance scores were obtained for this task. The mis- 
identification score was the number of words that belonged but were 
crossed out. The nonldentif Ication score was the number of words that did 
not belong but were not crossed out. 

The cloze test was also based on a passage written at a seventh-grade 
reading level. Every tenth word was deleted from the target passage and 
replaced by a blank, for a total of 50 deletions. The subjects were asked 
to fill in each blank with a word . consistent with the grammar and meaning 
of the surrounding text. The cloze-exact score was based on the subject's 
reproduction of the exact words deleted. The cloze-acceptable score was 
based on the subject's generation of words that were semantically and 
syntactically acceptable in the context. 

In the writing task the subjects wrote an essay on a topic selected 
from a number of possible topics. No time limit was imposed. Four 
readers evaluated each composition, and each reader assigned a score from 
1 (poor) to 9 (excellent) on each of four dimensions: structure, organi- 
zation, quantity, and vocabulary. The final score was the percent of 
total possible points a subject had earned, combined across all dimensions 
and readers. 
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Forty-thrce of the 54 subjects w»re «iainietered the interview test. 
The scoring procedures were siailsr to those used for the writing task, 
except that the four rating dimensions were comprehension, fluency, 
control over structure, and pronunciation* 



Results and Conclusions 



Table 1 displays the mean score, standard deviation, and sample size 
for each measure. 



Table 1 

Mean Scores and Standard Deviations for Editing Test, Cloze Test, 
TOEFL, and Composition and Interview Evaluations 



Measure 


Mean 


SD 


N 


Editing — Nonidentification ( # wrong) 


19.44 


13.62 


54 


Editing—Mi sidentifications ( # wrong) 


17.98 


14.20 


54 


Cloze-Acceptable ( # right) 


21.50 


10.45 


54 


Cloze-Exact < # right) 


14.90 


8.27 


54 


TOEFL Listening Comprehension 


47.27 


5.56 


18 


TOEFL Structure and Writtten Expression 


46.78 


7.35 


14 


TOEFL Reading Comprehension and Vocabulary 


48.21 


7.06 


14 


TOEFL Total 


468.40 


51.66 


22 


Composition — Z of possible points 


54.50 


21.98 


54 


Interview — X of possible points 


62.41 


22.30 


43 



Table 2 displays the correlations among the various measures. These 
cocrelations indicate th«t the nonidentification score on the editing test 
tended to correlate higher with the other neasures than did the misidenti- 
fication measure or a composite of the nonidentification and olsidentifi- 
cation measures. 
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Table 2 



Pearson Product-Moment Correlations of Scores on the Editing Test with the 
Cloze Test, TOEFL, and the Corapcsltlon and Interview Evaluations 



Measure 



Cloze-Exact 

Cloze-Acceptable 

TOEFL Listening Comprehension 

TOEFL St rue. & Written Express. 

TOEFL Rdg. Comp. & Voc. 

TOEFL Total 

Composition 

Interview 



Score on Editing lei 



Mlsldent. 


^Soalde^t . 


Mlsldent. + 
Nonldent . 


N 


-.25* 


-.73*** 


-.58*** 


54 


-.39** 


-.85*** 


-.74*** 


54 


-.32 


-.60** 


-.53* 


18 


-,55** 


-.60** 


-.64** 


14 


-.05 


- .16 




14 


-.25 


-.43* 


-.43* 


22 


-.39** 


-.82*** 


-.71*** 


54 


-.15 


-.7A*** 


-.53*** 


43 



*p < .05 



**p < .01 



***p < .001 



Interestingly, the number of Insertions correctly identified on the 
editing test (i.e., the complement of the nonldentlf icatlon score) corre- 
lated higher with the composition score (.82) and with the Interview score 
(.74) than did the cloze-acceptable score. (Corresponding correlations 
for the cloze-acceptable score were .77 and .67.) An Inspection of item 
facility and discrimination Indices showed that the two scores on the 
editing test had more desirable item characteristics than did the two 
scores on the cloze test. The Interltem reliability of the number of 
insertions correctly Identified on the editing test (.96) was higher than 
the corresponding reliability of the cloze-acceptable score (.92). 

In summary, the data suggest that the nonldentlf Icatlon score on the 
editing test was superior to the misldentlf icatlon score or the combina- 
tion of the two, and was superior to the cloze-acceptable score, as 
measured by correlations with the direct indices of English proficiency. 
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54. Mullen, K A* (1979) • ^tore on cloze tests as tests of proficiency in 
EngliRh as a second language^ In J* Brlere & F« Hinofotis 
(Eds.)t Concepts in language testing; Some recent studies 
(pp« 21-32) • Washington, DC: Teachers of English to Speakers 
of Other Languages. 



Purpose 



This study investigated performance on a cloze reading test as a 
function of cloze passage difficulty and method of scoring. It also 
investigated the criterion validity of cloze test performance in relation 
to performance on an oral interview, TOEFL, and a composition task* 



Method 



There were 154 subjects in the study, all of whom were administered 
an oral interview, a writing proficiency (i.e., composition) test, and a 
cloze test. TOEFL scores were available for 80 of the 154 stibjects [the 
source of the subjects and whether the three- or five-part TOEFL was used 
are not indicated]. 

The oral interview was 15 minutes in length, and 15 pairs of inter- 
viewers were involved. Interviewers first asked general questions related 
to the subject's background and future plans, then asked more specific 
questions* The two Interviewers Judged the subject's speaking proficiency 
on four scales: listening comprehension, pronunciation, fluency, and 
grammar. Scores ranged frc^ one to nine on each scale. 

The cloze test was administered following the oral interview. This 
test consisted of two passages, one classified as easy, the other as 
difficult, drawn from reading instruction material. The easy passage was 
Identified as a seventh-grade level text, and the difficult passage, a 
twelfth-grade level text. The cloze versions of the passages were con^ 
strutted by deleting ev^*" ' tenth word after the first paragraph, for a 
total of 50 deletions. Order of presentation of the easy and difficult 
cloze passages was counterbalanced across subjects. One hour was allotted 
for completion of the entire test. 

The composition test was administered after the cloze test* The 
subjects were given a booklet and were allowed one hour to write on an 
unspecified topic; they were told that their ccrapositions would be evalu^ 
ated on the basis of accurate sentence construction, appropriate use of 
vocabulary, organization of ideas, and quantity of writing. Compositions 
were scored by pairs of judges randomly selected from among eight pairs. 
Scores consisted of ordinal ratings on control over English structure, 
organization of ideas, quantity of writing, and appropriateness of vocabu-- 
iary . 
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The data for analysis coasisted of the Interview score > the coaposi- 
tion score, and the TOEFL score. Perforaance on the cloze test was 
described In terms of four neasures: easy/exact score, easy/ac-eptable 
score, difficult/ exact score, and difficult/acceptable score. These 
terns referred to the difficulty level of the passage and whether the 
subject was scored according to reproduction of the original word deleted 
(exact score) or a seaantlcally acceptable response (acceptable metlK>d) . 



Results and Conclusions 



Analysis of variance indicated that cloze test performance was most 
affected by individual differences among subjects and, to a anich lesser 
degree, by method of cloze scoring, passage difficulty, and tliB order in 
which the passages were presented. Method of cloze passage scoring 
accounted for more variance In scores than did level of passage diffi- 
culty, despite the fact that the easy passage «n8 rated at the seventh- 
grade level and the other passage was rated at the twelfth-grade level* 

The correlation between exact-word cloze score and oral interview 
score was not significantly different from the correlation between accept- 
able-word cloze score and oral Interview score. This was true for both 
easy and difficult cloze passages. The highest of these four correlations 
was that involving the acceptable-word score for the easy passage (£ ~ 
.57). 

The correlation of exact-word cloze score with TOEFL score was not 
significantly different from the correlation of acceptable-word cloze 
score with TOEFL score. This flndl.ig applied regardless of cloze passage 
difficulty level. The highest of these correlations was between the 
acceptable-word cloze score for the easy passage and TOEFL score (£ - 
.69), which accounted for 47.8 percent of the variance in TOEFL scores. 

,/ 

The pattern of correlations involving the compositions wr - somewhat 
different. For the difficult (but not the easy) cloze passage, *. *e accept- 
able-word cloze score correlated significantly higher with composition 
score than did exact-word score. Acceptable-word scoring of the easy 
cloze passage produced the highest correlation with composition test score 
(.76), predicting 58 percent of the variance in composition scores. In 
summary, the easy cloze passage score was the best predictor of scores on 
all three criterion measures, and this effect was most prominent for 
acceptable-word scores. 

Inspection of the reliability coefficients for the various measures 
Indicated that the correlation between a given cloze measure and a given 
proficiency test score was always less than the reliability of the test 
scores Involved in the correlation. The reliability data indicated that 
each criterion score measured some aspect of second language proficiency 
that was not measured by the cloze test scores. This is shown by the fact 
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that the percentage of reliable variance in the interview that vas not 
predicted by the cloze test ranged fron 58 percent for easy/acceptable 
score to 69 percent for difficult/exact score; analogous percentages for 
TOEFL ranged fro« 45 percent to 62 percent and, for the conposition, from 
32 percent to S3 percent* 

In general, it appears that the predictive ability of various clore 
test scores was dependent on whether speaking or writing was the criterion 
measure. 
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55. Odunzc, 0. J. <1982), Test of Engllch as a Foreign Language and 
first year CPA of Nigerian students (Doctoral dissertation, 
^ University of Missouri— Columbia, 1980), Dissertatioa Abstracts 
International , 42, 3419A-3420A. (University Microfilms no. 
8202657) 



Purpose 



The objective of this study was to determine the relationship between 
TOEFL and the academic performance of Nigerian students in four U.S. 
universities. The study also sought to determine whether TOEFL scores 
differ as a function of various background variables. 



Method 



Questionnaires soliciting participation were sent to 289 Nigerian 
students in four Missouri universities, and questionnaires were t^«twr«ed 
by 260 students. A total of 220 respondents were undergraduates, and 118 
of them had taken either the three-part or the five-part TOEFL between 
fall 1975 and winter 1980 [presumably via International or Special Center 
administrations) (none of the graduate students had taken TOEFL). These 
118 students comprised the sample for the study. Three of the four major 
ethnic groups of Nigeria were represented: Ibo (40 subjects), Yoruba 
(40), and Eflk (38); no Hausa students had taken TOEFL. 

Grade-point averages (CPAs) were available for all subjects, sepa- 
rately for the first semester and the second semester at their J.S. 
universities. 

Background information was also obtained from the students' question- 
naires, including (a) ethnic group, (b) type of secondary school ft^nded, 
(c) performance on the West African School Certificate examinations, (d) 
sex, and (e) parents' education. 



Results and Conclusions 



Correlational analyses showed no significant relationship between 
TOEFL scores and either first-semester CPA (r - .05) or second-semester 
CPA (r - .00). Several analyses of variance were also computed, to 
determine whether TOEFL scores differed as a function of various back- 
ground variables. These analyses revealed no significant differences in 
TOEFL scores (a) among the three ethnic groups; (b) between subjects 
passing the West African School Certificate examination in division II and 
those passing in division III; (c) between males and females; (d) between 
students from government and private secondary schools; (e) between 




•tudents fron Catholic «nd Anglican private schools; or (f) between 
students whose level of parental education was elementary, high school, 
university, or none. 

Analyses of variance also showed that students from Catholic, Angli- 
can, snd government secondary schools did not differ in first-semester CPA 
I analysis of second-scaester CPA is not reported] and that neither first- 
nor second-seaester CPA vsried as a function of ethnic group. 

Analyses w^re also conducted involving TOEFL Listening Comprehension 
and English Structure subscores. ["English Structure," for those taking 
the three-part TOEFL, is assumed to mean the Structure and Written 
Expression section. Analyses involving a "Vocabulary" score are also 
reported, but a separate vocabulary score cannot have been available for 
students taking the three-part TOEFL, so these analyses are not summarized 
here as their meaning is not clear.] The correlation between Listening 
Comprehension and f irst-s«sester CPA was significant U » .26). English 
Structure was not significantly correlated with first-sanester CPA (r - 
.04). [Comparable analyses for second-semester CPA are not reported.] 

It is concluded that TOEFL is not a good indicator of academic 
success in college. Although there was a small, significant relation 
between Listening Comprehension and academic performance, the replica- 
billty of this result needs to be determined. 
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56. Oiler, J. W., Jr., & Hinofotls, P. (1980) • Two autually exclusive 
hypothesee about eecond leoguege ability: Indivisible or 
partially divisible coapetence. In J. V» Oiler, Jr., & K. 
Perkins (2ds.), teseagch in language testing (pp. 13-23). 
Rowley, MA: Newbury House. (BRIC Document Reproduction Service 
No. ED 139 267). 



Purpose 



To gather evidence that would support either a unitary or a divisible 
coapetence hypothesis about the nature of second language proficiency, 
interrelationships aaong aeasiires i«re explori^. 



Method 

Test data were gathered froa two groups. The first group was com- 
posed of 159 students at the University off Tehran in Iran. With the help 
of the university and the Aaerican Field Service, all of these subjects 
were administered tte five-section TOEFL plus a cloze test and a dictation 
test during 1972 and 1973. The seven aeasures mre intercorrelated and a 
factor analysis was performed on tlM data. 

The second group was composed of 106 students at the Center for 
English as a Second Language (CESL) at Southern Illinois University (SIU). 
Ml of these subjects took a cloze test, the three-part (Listening Compre- 
hension, Structure, and Reading) CESL Placement Examination, and the 
Foreign Service Institute (FSI) oral proficiency interview, which produced 
five scores: accent, grsaasr, vocabulary, fluency, and comprehension. A 
subgroup of these students (N - 51) took the five-section TOEFL; these 
students generally fell into^tbe top half of the sample with respect to 
English proficiency. |The nature of the test administration is not 
indicated.] Intercorrelations were computed among the nine measures for 
the larger group (and among the 14 measures for those taking TOEFL); 
principal components factor analyses twre also perfon^d. 

Results 

The analysis for the Iranian students indicated that a single factor 
accounted for 100 percent of the total variance in the matrix. The 
product of the loadings of each pair of tests on the first or general (^) 
factor was almost identical to the correlation between the tests. This 
indicates that the general factor accounted for the correlation between 
each pair of measures. 
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Factor analysis for the SIU students Identified two factors that 
accounted for 100 percent of the total variance. The first factor ac- 
counted for 87 percent of the variance and received no loading less than 
69 percent on any single seasure. Thus, about 13 percent of the total 
variance was not accounted for by the ^ factor. Examination of the factor 
solution suggests the existence of a separate oral factor that explains 
the loading on the FSI interview. 

Among the subgroups that took TOEFL, there was considerably less 
variance on the CESL and FSI measures. In this case, the j| factor ac- 
counted for only 65 percent of the total variance on the three batteries 
of tests. Analysis of data for the subjects who took TOEFL indicated the 
possible existence of a third factor (listening), since heavy loadings on 
this factor were observed for both the TOEFL and CESL Listening Comprehen- 
sion sections. 



Conclusions 



Considering the results of all three sets of data, there is no 
support for the notion of separate components of structinre, vocabulary, 
and phonology. There is some evidence for a speaking factor associated 
with performance on the FSI interview. Overall, the evidence points to 
the existence of a general proficiency factor that accounts for 65 percent 
or more of the total variance in the several tests investigated. 
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57. Oiler, J. W., Jr., & Spolsky, B. (1979). The Teat of English at a 
Foreign Language. In B. Spolsky (Ed.), Some major teata. 
Advances in language tcatlng aeries: 1 . Papers in applied 
linguistics (pp. 92-100^. Arlington, VA: Center for Applied 
Linguistics. (Edited volume available as ERIC Document Repro- 
duction Service No. ED 183 004) 



Purpose 



This descriptive/analytic paper deals with the history of TOEFL's 
develoi»nent, assumptions underlying the teat, its use and interpretation, 
and future directions. [See also suamaries of Jameson & Malcolm, 1973, 
and Palmer, 1965, for details on the early history of TOEFL.] 



Discussion 



History of TOEFL 

At a 1961 conference on English language testing, the need was 
expressed for a systematic method of assessing the English proficiency of 
foreign applicants to U.S. colleges; this need was made further apparent 
by a 1962 survey of U.S. colleges, showing that several methods for 
evaluating applicants* English proficiency were in use at the time. On 
the basis of suggestions made at the 1961 conference, the National Council 
on the Testing of English as a Foreign Language was formed, representing 
30 organizations. With foundation support, the National Council estab- 
lished a staff to develop a test, to be named the Test of English as a 
Foreign Language, or TOEFL. The test was developed in 1963 and first 
administered in 1964. This objective test consisted of five parts las 
described in the Introduction to this collection.} 

Various procedural changes have taken place over the years. Until 
1969, all new questions were specially pretested with foreign students at 
American institutions; thereafter, all regular administrations in domestic 
test centers included experimental items to be included in later opera- 
tional tests. Also, local agents have taken over «iministrative functions 
where possible. Further, a new equating system was developed that would 
not require the reuse of items and thus jeopardize test security. Despite 
such procedural changes, however, the basic format of the test had not 
changed substantially at the time of this writing. 



Assumptions 

The following principles were set down to guide TOEFL development; 
(a) contrast ive analysis should not serve as a basis for test construc- 
tion, (b) testing of reading comprehension should not involve difficult 
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voc«bul«ry, and (c) language ahould be presented in a realistic context. 
In addition, test development also appeared to involve two implicit 
assuMptions: (a) the test should be standardized in relation to a large 
population of nonnative speakers, and (b) various skills should be tested, 
with the test containing both discrete-iten sections (English Structure, 
Vocabulary, and Writing Ability) and more integrative sections (Listening 
and Reading Coaprehension) . 

From a linguistic standpoint, these last two assunptions might be 
questioned. Regarding the first assiaption, one could argue that the test 
should be standardized in reference to native rather than nonnative 
speakers, particularly if the test is to be used to estimate readiness to 
study in the U.S. Angoff and Sharon (1971) found that, in the Writing 
Ability section of a certain TOEFL form» nonnative speakers outperformed 
native speakers on a full 21 percent of the Items. This would suggest 
that all TOEFL sections have not always focused on tasks that native 
speakers do well. « 

Regarding the second assumption, there is much research to suggest 
that the subscores on TOEFL or any other language proficiency test tend to 
be highly intercorrelated. Alttough the TOEFL manual (Educational Testing 
Service, 1973) reports low correlations between Listening Comprehension 
and other parts of the test, for most nonnative speakers the ability to 
listen with comprehension is highly related to reading and writing skills, 
as shown in research with the cloze test, dictation, and other measures. 
Perhaps the low correlations between TOEFL subscores reflect low validity 
of one or more sections. Of course, the notion of testing separate 
subskills predated much of the current research, and It is hoped that 
future test development will take account of this and later research. 

Use and Interpretation of TOEFL 

An issue of debate concerns prediction of a foreign student's academ- 
ic success on the basis of scores on TOEFL (or other English proficiency 
tests). Some scholars have argued that grade-point average (CPA) is not a 
relevant criterion for validating TOEFL, while others maintain that CPA 
should be predicted by TOEFL, The 1968 TOEFL manual^ refeired to several 
studies claiming to relate TOEFL to CPA as a criterion for validation. 
However, the 1973 manual (Educational Testing Service, 1973) noted that 
such studies "have generally yielded positive correlations... but... 
usually so low as to be of little practical usefulness in the admission 
process." The manual notes that this result is to be expected if TOEFL 
scores are used appropriately. If only those students with high English 
proficiency are admitted, or if course loads and English training are 
adjusted to meet the student's needs, English proficiency should not 
be a major factor in determining academic success. 



^Educational Testing Service (1968). Test of English as a Foreign 
Language; Interpretive information . Princeton, NJ: Author. 
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Another issue often raised is that TOEFL scores should not be inter- 
preted in an absolute manner and that a TOEFL score should be regarded as 
falling within a certain range of proficiency. The TOEFL interpretive 
manuals have sought to discourage absolute Interpretations and the use of 
cutoff scores. The 1968 manual urged institutions to conduct their own 
studies; at the same time, however, it did provide guidelines for admis- 
sions decisions based on various TOEFL score ranges, as suggested by some 
colleges* experiences. The later, 1973 manual, was more cautious, as it 
did not provide guidelines but, rather, presented the results of a 1969 
survey showing the TOEFL score ranges associated with various decisions at 
a number of Institutions. It also stressed several principles for the 
proper interpretation of TOEFL scores Isee Sumnary No. 27, Educational 
Testing Service, 1973]. 



Future Directions for the TOEFL 

At the time of writing of this paper [probably ca. 1974), changes in 
the TOEFL are under consideration, based partly on data from the study by 
Pike (1979), and partly on input from specialists. It has been difficult 
to say exactly what the five sections of TOEFL are measuring, and data 
from studies such as that of Pike will be helpful in addressing this 
issue. The Pike data show that, surprisingly. Listening Comprehension 
correlated better with an essay than with an interview score, and Writing 
Ability correlated no better with the essay test than with the interview 
and a cloze test. In general, Pike's data suggest that all the TOEFL 
subtests may be highly interrelated, so there appears to be little basis 
for concluding that the various test sections are measuring different 
skills. This view coincides, in part, with the recommendation from the 
Pike study that the number of test sections be reduced, with consolidation 
of those that are particularly highly correlated. 

It Is hoped that the changes under consideration will help improve 
the test's effectiveness as a measure of English proficiency, just as 
recent changes have improved the efficiency of TOEFL's administration. 
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58. Osapyinbi, J. A. (1975). A cooeurrent validity itudy of the U«8t 



African School Certificate and General Certificate of Education 
Engliih Language Examination, uaing Educational Testing Ser- 
vice's Test of English as a Foreign Language as the criterion 
measure (Doctoral dissertation* University of Wisconsin, 1974). 
Dissertation Abstracts International . 35, 5130A-5131A. (Univer- 
Bity HicrofilBs No. ih-il, iU) 



This study detenained the relationships oiong TOEFL and the part and 
total scores on the West African Examinations Council School Certificate 
and General Certificate of Education (SC/GCE) English Language Exoaination 
for a sample of the November 1972 SC/GCE candidate population in Nigeria. 
The SC/GCE English part and total scores were treated as predictor vari- 
ables, to be correlated with a set of criterion iseasures, TOEFL and the 
SC/GCE aggregate score (defined below). The objective was to obtain 
Infonsatlon that could be used to improve the validity of the SC/GCE 
English test. 



The subjects were 217 male and feaale graduating high school students 
in Nigeria in the fall of 1972. All students enrolled in five high 
schools randomly selected from 30 high schools located near a TOEFL test 
center in the state of Lagos served as subjects. The subjects were 
administered the SC/GCE general achievement and English language tests 
through standard operational testing procedures. They were also adminis- 
tered the five-part TOEFL in a special experimental session. One-hundred- 
sixty students also took the optional Oral English Test of the SC/GCE test 
battery. 

The SC/GCE English test consists of four subtests: two composition 
tasks, a test of reading comprehension and summary writing, and a test of 
vocabulary and syntax. The first subtest, labeled lA, is a 45-minute 
analytically scored essay written on a common topic, about which substan- 
tial information is provided. Subtest 2A is an additional measure of 
writing lasting 45 minutes. Usually four essay topics are given in 
Subtest 2A, with much less information provided for the examinee's use 
than on the first essay; any one of the four topics may be attempted. 
Subtest IB is a 75-minute test of reading comprehension and summary 
writing that requires the examinee to read three prose passages and then 
write a precis on one passage and give written answers to questions on the 
other two. Subtest 2B is a lOO-ltcm multiple-choice test of vocabulary 
and syntax; the 50-item vocabulary index is entitled Lexis (2B-L). and the 
50-it«a syntax index is called Structure (2B-S). The battery also includes 
an optional Oral English Test that consists of a iSO-wrd passage to be 
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read Aloud; a Xistenliig coaprebentlon measure In which the exanloee Is 
required to identify significant English stress, intonation, and phonemic 
contrasts in words and sentences; and an oral co«po»ition measure in which 
the examinee speaks for two minutes on a single topic chosen from a list 
of five topics. 

The SC/GCE examinations also include separate achievement tests 
selected by the student In 54 si&ject areas; the SC/GCE aggregate score is 
the sum of the six best scores on the SC/GCE subject tests. 



The mean TOEFL score of the subjects was 54?, Indicating that the 
group was above average in comparison with the general population of 
exsBinees taking TOEFL. The KR-20 reliability of TOEFL for this group was 
.973, which compares favorably with the average reliability of .965 cited 
in the 1973 TOEFL manual (Educational Testing Service, 1973). The inter- 
correlations between the two test batteries are presented in Table 1. 



Results 



Table 1 



TOEFL - SC/GCE Intercorrelations 



Variable 



Variable 



1 2 3 4 : 6 7 8 9 10 11 12 13 14 15 



1 SC/GCE lA 

2 SC/GCE 2A 

3 SC/GCE lA & 2A 

4 SC/GCE IB 

5 SC/GCE 2B-L 

6 SC/GCE 2B-S 

7 SC/GCE 2B-L-f2B-S 
i\ SC/GCE Eng. tot. 
9 SC/GCE Oral Eng. 

ID SC/GCE Aggregate 

11 TOEFL LC 

12 TOEFL ES 

13 TOEFL V 

14 TOEFL RC 

15 TOEFL WA 

16 TOEFL Total 



57 
85 
42 
21 
53 
36 
66 
28 
42 
32 
53 
35 
30 
46 
44 



91 
47 
25 
50 
39 
72 
23 
45 
39 
52 
37 
32 
50 
48 



50 
26 
58 
42 
78 
28 
49 
41 
59 
41 

35 
55 
52 



43 
61 
54 
78 
30 
53 
53 
61 
55 
56 
59 
65 



60 
91 
68 
26 
60 
62 
56 
73 
65 
65 
74 



87 
85 
46 
63 
66 
84 
73 
62 
81 
83 



84 
38 
67 
72 
76 
82 
73 
80 
87 



41 
70 
70 
83 
76 
68 

81 
86 



22 
55 
58 
42 
29 
37 
54 



51 
58 
63 
62 
62 
67 



75 
69 
62 
67 
88 



78 

61 75 

70 76 61 

90 90 81 86 



Note ; The decimal point has been omitted from each cf these correlations 
to save space. 
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A principal components factor analysis was applied to the intercorte- 
latlons shown above. This analysis identified three factors that accounted 
tor 76.4 percent of the variance. A general language ability factor 
accounted for 38,9 percent of the variance on both tests. The TOEFL 
scores showed heavier loadings on this factor than did the SC/GCE scores. 
An "essay" factor accounted for 21.5 percent of the variance. The SC/GCE 
scores loaded more heavily on this factor than did any of the TOEFL 
scores. Only the TOEFL English Structure and Writing Ability sections 
showed a moderate correlation with the essay factor. An oral English 
factor accounted for an additional 6 percent of the t«tal variance, 
although only the SC/GCE Oral English test showed a strong relationship 
(.93) to this factor. Among the TOEFL sections. Listening Comprehension 
showed the highest relationship (.56) with the oral English factor. 
English Structure was the only section to load significantly on all three 
factors, which probably explains why it was the best predictor of overall 
proficiency as measured by SC/GCE English total (see correlational data 
above). The Vocabulary section loaded heavily on Che general factor and 
only slightly on the oral English factor? the Reading Coi^rehension 
section ^ <■ ; n the general factor only. Ae Vhriting AbllltLy section 
loaded . . ly on the general language ability factor than on the 

essay 1 ^ iv s presenting only slight evidence of its construct 

validit> y-i ogate direct measure of essay writing skill. 



Conclusions 



The two SC/GCE ess«y subtests did not correlate well with TOEFL 
Writing Abilit^y. They showed low correlations with overall ESL competence 
and only a moderate correlation with each other, which is probably a 
result of their lack of reliability. Thus, their replacement by a multi- 
ple-choice measure of witing ability should be considered. The subtest 
measuring reading comprehension and simunary wr.ting should be replaced by 
a single multiple-choice measure of reading cwoprehenslon. Because this 
subtest confuses receptive and productive skills,.!^ does not predict 
TOEFL Reading Comprehension and shows Its highest correlations with TOEFL 
English Structure and Writing AMlity. The Lexis and Structure subtest 
was found to be a valid measure!. The Lexis score from this subtest 
correlated highest with TOEFL Vocabulary, while the Structure score 
correlated highest with TOEFL English- Structure. The relationship of 
English language test scored (TOEFL and SC/GCE) to the SC/GCE aggregate 
score shows the extent to which achievement in other subjects in West 
African high schools is related to English proficiency. 
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59. Pack, A. C. (1972). A comparison between TOEFL and Michigan test 
scores and student success in (1) freshman English and (2) 
cc^pleting a college progrma. TESL Reporter , 5^, 1-7, 9. 



Purpose 

This study examined (a) the relationship between the Michigan test 
and TOEFL, (b) the relationship between these tests for different language 
groups, and (c) the predictive validity of the tests for students at the 
Church College of Hawaii. 



Method 



A search was conducted of the records of students who attended the 
college between September 1960 and February 19^2. Scores on the five-part 
TOEFL [presumably obtained via International administrations], the Michi- 
gan test, and freshman English grades were available for 402 students. 
The Michigan test was the battery of tests including the Test of Aural 
Comprehension and the three-part Michigan Test of English Language Profi- 
ciency (MTELP), which tests grammar, vocabulary and reading comprehension. 
(For a more complete description of the MTELP, see Summary No. 1, Abadzi, 
1976. 1 For the total group the mean TOEFL score was 435 while the mean 
score on the MTELP was 67. Twenty-three members of this group graduated 
during this period. 



Results 



Correlations between TOEFL and the Michigan test were obtained. The 
correlations between equivalent sections on both tests were Aural or 
Listening Comprehension, .4 5; Structure-Grammar, .52;, Vocabulary, .62; 
Reading Comprehension, .49; and total test score, .66. [Note that the 
total score for the Michigan test used here is the total on the MTELP, 
which excludes the aural test.] Correlations were also obtained between 
total scores on these two tests for subjects from seven native countries. 
The correlations were Samoa, .55; Tonga, .69; Japan, .21; Taiwan, .66; 
Fiji, .80; Korea, .39; and Tahiti, .97. (Ns are not indicated.] 

Each test showed a significant positive correlation (£ < .001) with 
first-semester English grade. The test scores were not related to the 
probability that an examinee would graduate. [The obtained correlations 
and Ns are not given; also, the way in which probability of graduation was 
examined is not indicated. ] 
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Concluelons 



The correlation between TOEFL and the Michigan teet waa only modet' 
ate. Thua, theac two teata arc not interchangeable. While TOEFL and 
Michigan teat acorea are aignifleantly related to the grade obtained in 
the firat Engllah course taken, they are not related to grlKlea obtained in 
aubaequent Engliah couraea nor are they related to the probability that an 
examinee will graduate. 
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60. Palmer, L. A. (1965), TOEFL: Testing of English as a Foreign 
Language, Bulletin of the National Catholic Education Associa- 
tion , 62,, 235-238, 



Purpose 

This is a descriptive paper that details the background and early 
development of TOEFL [see also summaries of Oiler & Spolsky, 1979, and 
Jameson & Malcolm, 1973]. 



Discussion 



In May 1961, a conference on testing was held, sponsored by the 
National Association for Foreign Student Affairs, the Institute of Inter- 
national education, and the Center for Applied Linguistics. Participating 
in the conference were members of the sponsoring organizations, government 
agencies, and professional associations involved with admission of foreign 
students, as well as organizations concerned with the testing of English 
as a foreign language. The purpose of the conference was to consider how 
to evaluate the English proficiency of foreign applicants to U.S. colleges 
and universities. 

Despite the growing number of foreign applicants, English-language 
training facilities were inadequate at most institutions in which foreign 
students were enrolled. Furthermore, methods of determining these appli- 
cants' English competence prior to admission were quite varied. A U.S. 
Department of State survey of 950 Institutions of higher education that 
enrolled foreign students revealed four types of criteria in use: (a) 
English test score (approximately 25 percent of institutions); (b) certi- 
fication by a local official or English teacher, or an interview with a 
representative of the U.S. college or university (about 35 percent); (c) 
the applicant's school record (about 15 percent); and (d) the applicant's 
self-evaluation or correspondence in English (10 percent). More than 100 
institutions indicated that they required no evidence of English profi- 
ciency . 

Since the second of the above-mentioned categories, certification by 
a government off leal, often involved testing, the majority of applicants 
were likely given some type of test. Still, there were serious problems 
with the tests and testing practices in use. Test security was a major 
problen, as tests were being used repeatedly under nonstandard conditions, 
so that teat results could not be interpreted with confidence. Even less 
confidence could be placed in students' self-evaluations, or in school 
transcripts, which provide no direct evidence of English proficiency. 
Also, interviews tended to be unreliable, as the typical four-point rating 
scale allowed the rater nwch flexibility in interpretation. 
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Particlpants at the 1961 conference unanimously agreed on the need 
for a comprehensive testing program to evaluate the English proficiency of 
foreign applicants. Almost 95 percent of over 500 colleges and universi- 
ties surveyed in 1962 also indicated a need for such a program. 

As a result of the 1961 conference, the National Council on the 
Testing of English as a Foreign Language was formed in January 1962. The 
following year, with support by foundations to the ^todern Language Associ- 
ation of America, the National Council established a program office at the 
Center for Applied Linguistics in order to develop the proposed test, 
which became known as the Test of English as a Foreign Language, or TOEFL. 
A contract was signed with Educational Testing Service to administer and 
score the test and report the test scores. 

Development and administration of the test followed the guidelines 
proposed by the 1901 testing conference: (a) the test would be given 
under secure conditions on a worldwide basis three times a year (on the 
same dates at all te?t centers), with a new form developed for each 
administration; (b) the tests would be written by experienced teachers in 
English 38 a foreign language; (c) the policy-making body for the test 
would consist of representatives of government agencies and professional 
organizations involved with language assessment; and (d) the test would 
provide, along with a total score, reliable subscores indicating an 
examinee's listening comprehension, reading comprehension, knowledge of 
grammatical structure, command of English vocabulary, and recognition 
of appropriate style and usage in formal writing. The test that was 
deveJoped consisted of five sections Iwee Introduction to this collection] 
providing subscores in the five areas specified above. 

The test was first administered in February 1964 and again in Novem- 
ber 1964 and January 1965, which was the last atlminlstration as of this 
writing. The tests were given in 107 cities in 59 countries (including 13 
centers in the United States), and examinees included speakers of 71 
different languages. More than 100 schools and institutions have made use 
of the services of the TOEFL program. Although the total number of 
candidates for the first three administrations was relatively small 
(1,800), the number is expected to increase greatly in the coming years. 
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61. Perkins, K. , & Pharls, K. (1980). TOEFL scores in relation to 
standardized reading tests. In J. W. Oiler, Jr. , & K. Perkins 
(F.dsOf Research in languai^e testing . Rowley, MA: Newbury 
House. 

Purpose 

The issue under research was th& extent of association between the 
TOEFL score and scores on three standardlzei reading tests. It was 
hypothesized that there should be a moderate to strong correlation between 
the TOEFL score and reading measures among foreign students who were 
almost ready for college study in English. Reading was selected as an 
area for investigation due to its importance in the college curriculum and 
the fact that nearly all proficiency testing in English as a second 
language requires that the examinee possess dlBcourse reading skills. 



Method 

The subjecte were three groups of students enrolled in fall 1976 or 
spring 1977 at the Center for English as a Second Language (CESL) at 
Southern Illinois University, Carbondale. The subjects were all classi- 
fied In the top two (of six) levels of English proficiency based on 
TOEFL, Michigan, and CESL placonent test scores. 

Each of the three groups was given one of three English reading tests 
«*long with TOEFL [presumably the five-part version], which was adminis- 
tered routinely by the Center. Group 1 (N - 23) was administered the 
Nelson-Denny Reading Test, Form D, Intended for grades 9 through 16. This 
test contains subtests of Reading Rate, Comprehension, and Vocabidary. 
Only the Comprehension and Vocabulary subtests were used h<»re. Group 2 (jJ 
• 47) was administered level 2 of the Iowa Silent Reading Test; this level 
is Intended for grades 9 through 14. Only performance on the Vocabulary 
and Comprehension sections of this test were examined. The Vocabulary 
section consists of 50 items intended to measure the depth, breadth, and 
precision of the student's general reading vocabulary. The Comprehension 
section contains 50 items tapping literal detail, reasoning in reading, 
and evaluation of information read. Thirty-eight questions are based on 
six short passages, which the reader can reread while answering the 
questions. Twelve questions test short-term recall; each question follows 
a different passage and must be answered without looking back at the 
passage. Group 3 (N « 40) was administered the McGraw-Hill Basic Skills 
System Reading Test, which contains 10 short passages followed by a total 
of 30 questions te itlng for main ideas and supporting details. 



Results and Conclusions 

For Group 1 there was a correlation , of .49 •<£ < .05) between TOEFL 
score and the Nelson-Denny test score. The TOEFL Reading Comprehension 
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Bubscore correlated .15 with the Nelson-Denny score. For Group 2, 
the TOEFL total score correlated .46 <£ < .01) with the Iowa test score. 
The correlation between the TOEFL Reading Comprehension subscore and the 
Iowa test score was .23. For Group 3, only the TOEFL Reading Comprehen- 
sion subscore was used; this subscore correlated .91 < .01) with the 
McGrair-Ulll test score. 

It Is concluded that the TOEFL total score, as hypothesized, Is 
moderately to strongly related to reading test scores. Regarding the 
finding that TOEFL Reading Comprehension subscores did not correlate as 
well as expected with reading test scores for Groups 1 and 2, either of 
t*» explanations for this finding seems most likely. First, the number of 
Items comprising the TOEFL Reading Comprehension section is smaller than 
the number of Items on the whole TOEFL. This would reduce the variance of 
observed scores, and, as a result, the correlation between the TOEFL 
Reading Comprehension section and reading test scores might be expected to 
be smaller than the correlation between TOEFL total score and the reading 
measures. Second, the standardized reading measures used here might be 
speeded for nonnative English speakers. As a result, the construct 
validity of the reading tests might be questioned for nonnative English 
examinees. 
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62. Pike, L. W, (1979). An evaluation of alternative Item formats for 
testing English aa a foreign language . (TOEFL Research Rep. No. 
2; ETS Research Rep. No. 79-6.) Princeton, NJ: Educational 
Testing Service. (ERIC Document Reproduction Service No. ED 206 
627.) 



Purpose 



Data were collected that would contribute to evaluating the five-part 
TOEFL and deteraining ways in which the test might be revised. Correla- 
tions were examined between sections of the test and (a) direct measiires 
of speaking and writing ability, (b) indirect measures of writing ability, 
and (c) I xperimental nultiple-cholce tests. Students in three different 
countries <*ere included to determine the generality of the relationships 
observed. 



Method 



The subjects were 98 examinees from Peru, 145 from Chile, and 199 
from Japan. Scores on the five-part TOEFL were obtained from the October 
1971 International administration for all subjects in the Peruvian sample 
and for about half the subjects in the other groups; scores were obtained 
from a special administration of the same TOEFL form for the remaining 
subjects. The five TOEFL subtests are Listening Comprehension, English 
Structure, Vocabulary, Reading Comprehension, and Writing Ability [see 
Introduction to summaries]. 

Other instruments were administered a short time after the adminis- 
tration of TOEFL. The subjects* scores on an essay test and an interview 
were obtained, to provide direct evidence of writing and speaking ability. 
[Both essay form and essay content were assessed, although essay form was 
of principal interest in this study.] The cloze test, in which the 
subjects determined the words deleted from a passage, also provided an 
index of writing ability that was more direct than a multiple-choice test 
but less direct than an essay (it is also thought to provide evidence of 
reading ability). A rewriting test, requiring the subjects to rewrite a 
passage of short, choppy sentences, provided another indirect measure of 
writing ability. Finally, four experimental objective tests were admin- 
istered. In the sentence comprehension test, the subjects read and 
answered questions about written questions or stateaents. In the words in 
context test, the subjects selected the word or phrase that best matched 
that underlined in a sentence. The combining sentences test required the 
subjects to choose the best cwnbination of several short sentences. The 
paragraph completion test was a multiple-choice cloze test; for each blank 
in a passage, the subject was to choose the deleted word from among four 
choices. 
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Results and Conclusions 



Mean scores on the nine objective oeasures wre lowest for subjects 
from Perut while scores for subjects from Chile and Japan were relatively 
comparable to each other» with some variation across measures. Mean 
scores on the other measures mte lowest for Peru and highest for Chile* 
These differences may reflect social class differences in the samples* 
Reliabilities of the measures (both objective and subjective) were 
highest for Peru (.82 to .98). most likely due to the fact that there was 
greater variation in scores for this group; in most cases » reliabilities 
were lowest for Japan (.66 to .92)* 

Intertest relationships were examined by correlations corrected 
for attenuation (unreliability)* The Listening Comprehension subtest 
showed lower corrected correlations with the eight other objective mea- 
sures than did the latter tests among themselves* Thus, the listening 
isk apparently contributed variance not contributed by tests presented in 
the written mode* Reasonably high corrected correlations betwen Listening 
Comprehension and interview scores (ranging from *75 to *84 across groups) 
suggest that Listening Comprehension is an effective estimator of spoken 
communication ability* These results point to the value of retaining 
Listening Comprehension as a separate measure in TOEFL. 

English Structure generally ranked second among the nine objective 
tests in reliability (*78 to *92 across countries)* It was usually the 
best estimator of the essay form score (corrected v^b of *81 to *98) and 
waft one of the tw best estimators of the interview scores (corrected jr's 
of .69 to .88). These results indicate the value of retaining the English 
Structure section In TOEFL* A high relationship between this section and 
Writing Ability (corrected jr's of *82 to .97) suggest that these two 
sections could be combined* 

The Vocabulary score showed a relatively high relation to Reading 
Comprehension (corrected jr*s of .88 to .95) suggesting that these two 
sections could be ccnabined* 

Reading Comprehension correlated highly with three of the four 
experimental tests: words in context, combining sentences, and paragraph 
completion (corrected £'s of .93 to .99) as well as with cloze test scores 
(.88 to .97). Thus, these other tests could also be used to tap processes 
related to reading ability. 

Writing Ability estimated the essay form score reasonably well, with 
corrected £'s of *93, *88, and *73 for Peru, Chile, and, Japan, respective- 
ly. Thus, for the first two populations at least, the data did not 
support the notion that the TOEFL Writing Ability section should be 
replaced with a writing sample* English Structure correlated highly with 
Writing Ability (corrected £^s of *82 to .97), suggesting that these two 
sections could be combined. Of the two subtests of Writing Ability, the 
one requiring error recognition correlated more highly with the essay form 
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•cor« (corr«et«4 jr*t of .75 to •93) than did tb« ono requiring sentence 
coapletlon (.55 to .84) » Implying that the former aectlon la more uaeftil 
aa an Indirect index of aubjecta' writing ability. 

Of the experimental aubteata, three abomed high corrected correla- 
tlona with Reading Coi^rehenaion, aa noted above. The fourth, the aentence 
coaprehenaion teat, ahomed lower corrected correlationa with Reading 
Coaprehenalon (.78 to .91), apparently becauae many aubjecta received 
perfect or near-perfect acorea on thia teat, thua reatrictlng the range of 
scorea* 

The acorea for the rewriting teat ahoved diaappointingly low correct- 
ed correlations with the easay form acorea (.19 to .54), perhapa becauae 
of the wide range of atructural complexity in the atd>Jecta' reaponaea. 
Indicating that the rewriting taak may be of limited value in if preaent 
fons. 

The close teat waa a reasonably good estimator of essay form acorea 
(corrected £'a of .78 to .94) and Reading Coaprehenalon (.88 to .97), 
indicating its value as an indirect meaaure of writing or reading ability. 
The cloze test, in turn, was well estimated by several multiple-choice 
tests, including the worda In context test (.94 to .98), the paragraph 
completion teat (.86 to .99), and TOEFL English Structure (.82 to .93). 

In general, the results suggest that the Llatenlng Comprehension 
scores are relatively independent of the other multiple-choice aeaaurea; 
English Structure and Writing Ability form one cluater, and Reading 
Comprehension and Vocsbulary form another. An Implication is that TOEFL 
could be revised to provide three scores, corresponding to these three 
apparent groupings. 
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63. Pitcher, B., & Ra, J. B. (1967). The reXatloa betweer acoret on the 
Test of English as a Foreign Language and ratings of actual 
theme wriTini (Statistical Rep. No. 67-9). Princeton, NJ: 
Educational Testing Service* 



Purpose 

This tudy sought to determine the degree to which students* writing 
ability c«n be measured with an objective test by assessing the relation 
between the quality of students* writing samples and their scores on 
TOEFL, particularly the Writing Ability section. 



Method 



The subjects were 310 foreign students who had recently arrived in 
the U.S. and were enrolled In university courses in English as a foreign 
language or in intensive English language courses at six different insti- 
tutions of higher education. 

Scores on the five-part TOEFL were available for each student (pre- 
sumably frcai International administrations]. Also, each student was given 
30 minutes to wlte each of four themes dealing with (a) a holiday or 
festival, (b) a family member, (c) a city, and (d) a significant event. 
Fourteen persons served as readers, and each theme was rated by two 
readers. After a practice session in which rating standards were estab- 
lished by examining 10 sample themes, the themes were rated on a four- 
point scale. During the readings, the readers occasionally discussed and 
rated "problem themes," to further ensure consistency of rating standards. 
A student's overall score was the sum of the two ratings on each of the 
four themes; scores thus ranged from 8 to 32. 



Results and Conclusions 



One possible method of estimating reliability of ratings would be to 
obtain a correlation between the two ratings of each theme, for all 1,240 
themes; this correlation was .64. Note, however, that this method ignores 
theme topics and counts each subject four times, A method that avoids 
these problems is to correlate the sum of the first fovu: ratings with the 
sum of the second four ratings, for the four themes written by each 
subject. The correlation obtained by this method was .85. Application of 
the Spearman-Brown correction to this figure yielded a reliability coeffi- 
cient of .92 for the sum of all eight ratings combined. 

Correlations with overall theme rating for the TOEFL part scores were 
Listening Comprehension, .56; English Structure, .74; Vocabulary, ,71; 
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Readlng Coaprehenslon, .65; and Writing Ability, •74. Correlation with 
total TOEFL score was «78. 

Correlations were computed among TOEFL subtests. The highest 
correlations were those among En^ llsh Structure, Vocabulary, and Writing 
Ability (ranging from .75 to .79); Listening Comprehension shoirad the 
lowest correlations with the other parts of the test. However, the 
range of correlations was relatively small — .60 to .79. 

Stepwise regression analyses were also performed In which theme 
ratings were predicted from various combinations of TOEFL part scores. 
This analysis shewed that the combination of English Structure and Writing 
Ability provided somewhat more effective prediction (multiple R « .78) 
than did either of these two subtests alone, and that prediction was 
improved slightly by also including the Vocabulary score (multiple R " 
.79). The other two TOEFL subtests are believed to measure aspects of 
language proficiency that are less Important In predicting theme writing. 
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64. Powers, D* (1980) • The relationship between scores on the Gradti^ 
ate Management Adaission Test and the Test of English as a 
Foreign Language (TOEFL Research Rep. !io> 5; ETS Research Rep. 
No. 8U-31). Princeton, NJ: Educational Testing Service. (ERIC 
Document Reproduction Service No. ED 218 329} 



^ Purpose 



This study Investigated the relationship between section and total 
scores on the Graduate Management Admission Test (GNAT) and TOEFL* Of 
particular Interest ms the role played by the examinee's native language. 
It wss hoped that the results would provide an explanation for the ccKunon-** 
ly observed ^^discrepancy** between TOEFL and OiAT scores. A second objec- 
tive was to Investigate the accuracy with which GMAT ex^lnees report 
their TOEFL scores » In order to determine If self-reported scores could be 
used In certain contexts. 



Method 



A sample of S^ySl GMAT candidates, who had also taken the three**psrt 
TOEFL via International or Special Center administrations, were drawn from 
test files for the period between September 1977 and August 1979. The 
GMAT, a multiple-choice test designed for applicants to graduate schools 
of business, measures general verbal and mathematical abilities. The 
verbal section measures ability to understand and evaluate uhat is read 
and to recognize basic conventions of standard written English. The 
quantitative section tests basic mathematical skills and imderstanding of 
elementary mathematical concepts, as well as the ability to reason quanti- 
tatively, to solve quantitative problems, and to interpret data given in 
graphs, charts, or tables ^description paraphrased frcm a recent GMAT 
publication] 

Only examinees who indicated that they were not U.S. citizens and 
that English was not their primary language were selected. The average 
TOEFL score for the group was 552, which is considerably greater than the 
average of 506 for all graduate-level TOEFL examinees during 1976-77. The 
mean GMAT total score of this group was 394.5 as compared with a GMAT 
population mean of 500. The mean verbal score on the (»IAT (OlAT-V) was 
16.7 as compared to 26 for the GMAT population, and the mean quantitative 
score (GMAT-Q) was 28.4 as compared to 27 for all GMAT candidates tested 
during 1975*^1978. Thus, these foreign candidates scored considerably 
below the mean on the GMAT-V and on the total test, and they scored 
slightly above the mean on the GMAT-KJ. 

Section and total scores on both tests were correlated using linear 
and curvilinear (quadratic) regression. The relationships were computed 
for all candidates and for the countries with the largest contingents* 
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The reported TOEFL scores of GNAT candidates were correlated with the 
actual TOEFL scores recorded In TOEFL progran files, and the relationships 
between those scores and other selected variables were compared. 



Results and Conclusions 



Table 1 shows simple linear correlations between GMAT and TOEFL 
scores. As can be seen, the correlations between TOEFL subscores and 
GMAT-V score are considerably higher (.58 to .69) than those between TOEFL 
subscores and CMAT-Q score (.29 to .39). Thus, while these aptitude test 
scores consistently correlated with proficiency in English, the correla- 
tion was much higher for aptitude In the verbal dcxaain than for aptitude 
in the quantitative domain. Table 1 also shows that the overall pattern 
of correlations of TOEFL scores with GMAT-V and GMAT-Q scores provides 
support for the discriminant validity of TOEFL as a measure of verbal 
rather than quantitative skills. The Listening Comprehension section 
showed a lower relationship with GMAT-V than did the other two TOEFL 
subscores. This finding is to be expected since the GMAT-V does not 
measure listening comprehension. 



Table 1 



Correlations Between GHAT and TOEFL Scores 



TOEFL Score 



GMAT Score 



Listening 
Comprehension 



Structure 
& 

Written 

Expression 



GMAT-Verbal .58 
GMAT-Quantitatlve .29 
GMAT-Total .52 



.66 
.37 
.61 



Reading 
Comp. 
& 

Vocabulary 



.69 
.39 
.64 



Total 

.71 
.39 
.66 



The use of curvilinear regression significantly Increased the corre- 
lations between TOEFL scores and ©lAT-V (Increases ranged from .02 to .05) 
but did not Increase the correlations with GMAT-Q. As would be expected, 
the Increase in correlations with GMAT-Total, which is a combination of 
the verbal and quantitative scores. Is midway between those for GMAT-V and 
GMAT-Q. 

Using a quadratic equation, the relationship between TOEFL and GMAT-V 
and total scores was determined for the five native language groups having 
the largest number of GMAT candidates (India, Iran, Japan, Taiwan, and 
Thailand). The relationship between GMAT-V and TOEFL scores was similar 
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across language K>^oups, excep^ that the correlations were lower for 
Iranian examinees. The Iranians received lower scores than did any other 
group on both tests, and because the GMAT scores should show little 
differentiation among very low scoring candidates, the correlations 
between GMAT and TOEFL scores would also be expected to be lower for this 
group. 

Finally, the mean section and total scores on both tests were calcu- 
lated for each of 137 different native country groups. The native country 
means on both tests were then correlated and each correlation was found to 
be .91 or higher* Thus, the discrepancy observed between the TOEFL and 
GMAT scores of foreign students was consistent from country to country and 
wa« »-plated to examinee English language proficiency. Whfle nonnative 
English speakers consistently score lower on the GMAT than do native 
speakers, exaoiinees who score higher on the TOEFL also tend to score 
higher on the GMAT. 

The correlation betwe ;n self-reported and actual TOEFL scores was 
found to oe .91 or .92 (depending on the regression method used). The 
correlations of self'-reported and actual TOEFL scores with GVJJ! scores and 
with self-reported undergraduate grade-point average were nearly Identl*- 
cal. This suggests that self-^reported TOEFL scores may be a useful 
substitute for actual TOEFL scores in studies in which the focus is on 
averages for groups or relationships among variables rather than on 
individual s. 



ERLC 



179 



-174- Bl, C2 



65. Ratchford, D. L. (1982). Keadi ability of entering freshmen 
International students at a Boutiiwestern state university: Some 
implications (Doctoral dlssertat ion. University of Oklahoma, 
1981). Dissertation Abstracts InternaMonal , 42, 3088A-3089A. 
(University Microfilms No. 8129435) 



Purpose 



The present study examined the reading ability of foreign students 
seeking enrollment at e southwestern state university. One objective was 
to determine how well students were prepared to read textbooks, where this 
assessment was based on the students' cloze test scores and experts' 
Judgments of the students' reading ability. A second objective was to 
assess the validity of scores on TOEFL, the Nelson-Denny Reading Test, and 
the cloze test against sti^lents' reading grade level as determined by the 
panel of expe^rts. 



Method 



The subjects were 70 foreign students beginning their freshman year 
at the University of Oklahoma. All subjects had scores on the three-^part 
TOEFL (presumably obtained in International or Special Center administra- 
tions j; only the original TOEFL score was used for any subject who had 
taken the test more than once. Studer.ts were sampled so that exactly 14 
subjects fell into each of five TOEFL score ranges: 400-449, 450-499, 
^00--549, 5 50-599 and 600-649. The sample included only students majoring 
In engineering or nonsclence related fields. 

The subjects were administered the Nelson-Denny Reading Test. This 
test, designed for grades 9 to 16, consists of iOO vocabulary questions 
and 36 reading comprehension questions abased, an eight reading passages) 
lequiring 10 minutes and 20 minutes, > espectively, to administer. The 
total score on this test is the score on the vocabulary items plus twice 
the score on the comprehension items. (Although the test also yields a 
reading rate score, this score was not used iii the present study). The 
subjects were also administered two clce tests. One was the general 
cloze test, which was constructed from a political silence text. The 
passages used in tnis test fell in the upper one-third of reading diffl** 
culty (as determined by the Dale-Chall Readability Formula) among a set of 
passages sampled from general core course textbooks encountered by under- 
graduates. The second cloze test was either the nonsclence cloze test 
(administered to 31 subjects) or the science cloze test (administered to 
39 subjects). The nonsclence cloze test used a passage from a journalism 
text, while the science cloze test used a passage from an engineering 
text. In constructing cloze test passages, every fifth word was deleted 
from the passages, which varied in length from 297 to 340 words. 



ERLC 



-ISO 



-175- 



A panel of reading experts assessed the general reading levels and 
reading level within the major field, for each subject. For this purpose, 
an instrument known as the Informal Reading Inventory was administered 
individuaVy to a subject by one of the panel members. The subject read a 
aeries of passages, selected so as to represent readability levels ra^girng 
from the fourth through the sixteenth grade, as determined by use of the 
Fry and Flesch Readability Formulas. After reading a passage, the subject 
wrote answers to a brief set of questions about the passage [the exact 
number of questions is not given] . The panel member then evaluated the 
subject's answers* If an answer was vague the panel member restated the 
question until a response was given that indicated understanding or lack 
of understanding of the passage. If a subject was found to exhibit 
at least a 75 percent comprehension of a passage (as determined by a 
criterion established in other research), he ox she was permitted to go on 
to the next most difficult passage. The total number of passages read by 
a subject was thus dependent on attaining a criterion level of perform-- 
ance. The final performance score assigned to the subjects was an 
Integer from 4 to 16 that represented their reading grade level. A 
comparison of reading level Judgments of the four panel members showed 
very high agreement for a sample of three subjects. 

Results and Conclusions 



Four sets or hypothesis were tested. Hypothesis 1 was that there 
would be a relation between the j?eneral cloze test score and the TOEFL 
score. Inspection of the data siiows i.h«*t, even though only four subjects 
answered 40 percent or more of the cJ'^ze iLc^zi? correctly (and thus wre 
defined as falling into the group i aslfled as the "instructional 
level"), the general cloze test scorp c related .71 with the TOEFL score 
{£ < .001). 

Hypothesis 2 was similar to Hypothesis 1, except that it involved the 
relation between TOEFL and either the nonsclence or science cloze test. 
For the nonscience cloze test, although only five of the 31 subjects 
scored at the instructional level, the score on this test correlated .78 
with TOEFL score (£ < .001). For the science cloze test, only three 
subjects scored at the instructional level, and the correlation with TOEFL 
score was •55 ij^ ^ .001). 

Hypothesis 3 was that the erpert panel*s judgment of the subject's 
general reading grade level would relate to the other measures in the 
8tudy---i.e . , scores on the TOEFL, cloze, and Nelson-Denny tests. The 
TOEFL Jcore was found to correlate .67 (£ < .001) with the panel's judg- 
ment of reading grade level as determined by the Informal Reading Inven- 
tory. Performance on the general cloze test correlated .61 (£ < .001) 
with judgment of th^^ subject's reading grade level. The total Nelson-Denny 
score correlated .68 with the subject's reading grade levsil {£ < .001) ; 
th^ correlation for the Nelson-Denny Vocabulary subscorc was .61 
(£ < .001) and the correlation for the Nelson-Denny Comprehension subscore 
was .62 (£ < .OOx). 
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Hy pot heels 4 investigated whether there was a relationship among the 
experts' judgment of the subject's reading grade level, Nelson-Denny 
Reading Test score, general cloze test score, and performance on the 
nonsclence or science cloze test of rearming ability. TOEFL scores were 
also reconsidered in the ensuing analyses. Inspection of the correlations 
between judged reading grade level and other scores for nonsclence oxajors 
showed no correlation higher than .65. (This latter correlation was 
established as a criterion for deciding that a given proficiency measure 
was an adequate indicator of a subject's reading ability.) A very similar 
result was found for science majors. While none of the correlations 
attained the .65 level, the correlations ranged from .45 to .63 for the 
nonsclence majors, with the highest correlation between Nelson-Denny total 
score and judged reading grade level. The corresponding correlations for 
science majors ranged only from .29 to .57, with the highest correlation 
between TOEFL and Judged reading grade level. 

In general, the TOEFL score and the Nelson-Denny Reading Test score 
appear to be useful for screening entering foreign students with regard to 
their reading ability. Perhaps the Nelson-Denny Reading Test could be 
used as a supplement to TOEFL in making decisions about these students' 
English reading proficiency. Further research could provide additional 
information on the validity of the Instruments used in the study. 



-177- 



Bi, Dl 



66* Rlgg8» J, M« (1982) # Close testing procedures in ESL: A prediction 



This study exatained the relation of first-year grade-point average 
(CPA) to cloze test scores and to TOEFL scores for foreign students at a 
coQiounlty college* 



Although the purpose of TOEFL Is not to predict academic success » and 
the evidence suggests that it is not very effective for this purpose, 
college faculty use TOEFL scores , in effect » to make decisions about 
probability of student success. Therefore, another test should be used, 
at least as a supplement to TOEFL, to determine a foreign student's 
ability to begin academic work. One possibility in this regard Is the 
cloze test, in which students determine the words that have been deleted 
fr m a prose passage. The cloze test is an integrative, functional test 
in that it measures general language proficiency in the context of si«cl- 
lie subject matter. TOEFL, in contrast, is a discrete-point type of test, 
which assesses knowledge of specific lexical, grammatical, or syntactical 
items. An Integrative test is believed to be more related to one's 
ability to handle language in a contextually rich environment and thue 
more suitable for predicting academic success. 



The subjects were 23 foreign students *^o registered in fall 1977 at 
Vlncennce University, a two--year community college in Indiana (seven 
additional students in the original sample were excluded due to transfer 
or placement into ESL classes). The large majority of ihe subjects (18 of 
the 23) were Arabic speakers and all but one were males. Students' can 
begin academic work £t Vincennes either by presenting a TOEFL score of 
at least 500 or by completing a set of courses in English as a second 
language. {The range of TOEFL scores reported below suggests that practi- 
cally all subjects %iere in the latter category.] 

The five-part TOEFL was administered before registration in fall 1977 
to all subjects. The^cioza test was administered within one week of the 
TOEFL. The cloze test consisted of three pai ages, two taken from a 
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general chemistry text used at Vincennes University and one from the 
commonly used freshman composition text. Every seventh word of each 
passage was deleted (except th#t the first and last sentence of eaifh 
passage remained intact). The passages contained a total of 148 blanks, 
with approximately the same niaaber of blanks per passage. The subject was 
to fill in the word that best completed each blank. All subjects finished 
well within the two hours alloted. The synonym, or acceptable-word, 
scoring method was used, in that synonyms for deleted words were accepted 
as correct; misspellings were not counted as mistakes. 

CPAs were obtained for the fall semester, for the spring semester, 
and for the combined two semesters of the first year for each subject. 



Results 



TOEFL scores for the 23 subjects ranged from 314 to 522, with a 
median of 374. Rel atl.>nships with GPA were examined by Spearman rank 
correlation. TOEFL score and cloze score, respectively, correlated .29 
and .5; with overall GPA, .33 and .67 with fall-semaster GPA, and .10 and 
,2 7 with spring-semester GPA. The correlation with overall GPA was 
nonsignificant for XOEFL but significant for the cloze test. 

\ separate analy&ls was done for subjects taking Composition 1 and 
chemistry, based on the combined grade for those courses alone not 
reported). The Spearman rank correlation with this score was .50 for 
TOEFL and .70 for the cloze test. 

The KR-21 reliability of the cloze test was .97. Reliabilities of 
the five parts of TOEFL ranged from .78 to .89. Scores on the cloze test 
con elated .77 with those on TOEFL. 



Conclusions 



The low correlation between TOEFL and GPA found here is consistent 
wit.i-the findings of some other rcudles. The small sample size makes it 
difficult to draw general conclusions, but the difference in correlations 
with TOEFL and the cloze test are worth noting. The suggestion is that a 
cloze test such as that used here more closely approximates a measure of 
the student's ability to perform in a language situation involving course- 
work than does TOEFL. The .77 correlation between the clw<;e test and 
TOEFL suggests some commonality between these tests, but the differential 
correlations with GPA suggest that the cloze test may be tapping ah 
additional factor, perhaps something resembling academic readiness. 

The fact that correlations with GPA were lower for the second semes- 
ter than the first may be due to improvement in students' English skills 
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during the first semester, which tends to reduce the role of language 
ability in deteraining scsdeaic perfomance. 

It is suggested that a test such as the cloze test be considered as a 
supplsnent to TOEFL in Baking adaissions decisions about foreign students. 
More extensive eapirical nork would be needed, however, to determine the 
predictive validity of the test for this purpose and to establish appro- 
priate de^cision scores for any specific institution that uould use ±tc 
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67. Scholz, G. E., & Scholz , C. M. (1981, Detroit). Multiple-choice 

cloze tests of EST discourse: An exploration . Paper presented 

at the fifteenth annual TESOL convention. (ERIC Document 
Reproduction Service No. ED 208 656) 



Purpose 

This study examined various types of cloze tests, including a stand- 
ard open-ended cloze test and several multiple-choice cloze tests that 
differed la the method by which distractors were selected. For groups of 
Chinese scientists, correlations were conqjuted between these tests and 
criterion measures, including selected subtests of TOEFL and of the 
Comprehensive English Language Test for Speakers of English as a Second 
Language (CELT) . 



Method 

The principal subjects were Chinese scientists attending an exchange 
program between Chin and the University of Californla—Los Angeles in the 
People's Republic of China. The subjects were drawn from many regions of 
China and ranged In age from 37 to 45. Most of them were lecturers, 
researcher:} , or professors In universities and technical Institutes in 
China. 

Two passages were used for the cloze test, one of 41!) words involving 
science as an academic subject, anj the other of 440 words involving 
science as a topic of popular Interest. In each passage, every seventh 
word was deleted, except that one or two sentences at the beginning and 
end remained Intact. There were 50 deletions per passage. 

For tne standard, opeg-ended cloze test , the subject was to write, In 
the blank space corresponding to each deleted word, the word that would 
best fill the space. Acceptable-word scoring was used; i.e., reasonable 
substitutes for the deleted word were accepted as correct. Four multiple- 
choice (MC) cloze tests were also developed. In one, f.he Interlingual MC 
cloze test , the distractors were the three most common errors made by a 
sample of 161 Algerian students of the English language; the fo jrth 
response alternative was the deleted word. 

To develop the other MC cloze tests, the two above-mentioned cloze 
tests were administered to a sample (Sample 1) of 187 Chinese scientists, 
the open-ended clozo test to 91 subjects, and the interlingual MC cloze 
test to 96 different subjects in the sample. The interlingual MC cloze 
ten was then item-analyzed, and items with facility less than .15 (i.e., 
fewer than 15 percent of the subjects answering them correctly) or greater 
than .85 wjre eliminated, as were items with a discrimination parameter of 
.25 or less (i.e., cor'-^lation of .25 or less with total test score). The 
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result was the revised Intsrllngual MC cloze test , which consisted of 20 
iteias* In addition^ the intrallngual MC cloze test was developed by 
using, as dlstractors for each deleted word, the three ©ost common errors 
made by the 91 Chinese subjects in Sample 1 who were given the open-tended 
cloze test* Finally , a teacher^-^made MC cloze te st was developed by having 
Chinese and American teact^rs of English as a second language, working in 
groups, write what they believed to be the best distractors for each 
deleted word. 

These last three MC cloze tests (revised interlingual, intralingual, 
and teacher-^made) were administered to three randomly selected groups 
drawn from a second sample (Sample 2) of 167 Chinese scientists (£s « 57, 
56, and 54, respectively, for the three types of cloze test.) Also 
administered to all subjects were the Structure and Listening subtests of 
the CELT and the Reading Comprehension, Vocabulary, and Writing Ability 
subtests of the five-part TOEFL. The combination of these five scores, 
here labeled "composition proficiency score," was also computed for each 
subject. 



Results and Conclusions 



Reliabilities of the TOEFL subtests ranged from .^5 to .81 for Sample 
1 (N - 187) and frooi ,60 to .90 for Sample 2 (hJ » 167), with the lowest 
reliability observed for Writing Ability and the highest for Vocabulary in 
each case. Reliabilities for the CELT subtests ranged from .52 to .78 
across samples, and reliabilities for composite proficiency scores were 
.87 and .90 for the two samples. The homogeneity of the samples may have 
contributed to relatively low reliabilities for some subtests. 

Reliabilities of the various versions of the cloze test, excluding 
the revised interlingual MC cloze test, were relatively consistent, 
ranging from .52 to .68. (For the revised Interlingual MC cloze test, 
which had 20 rather than the full 50 It^s, reliabilities were .67 and .48 
for the academic and popular passages, respectively.) Although these 
reliabilities are somewhat lower than those observed in previous studies, 
this finding may have been due to the homogeneity of the samples. 

Data pertaining to the criterion validity of the cloze tests are 
available in the correlatinns between each of the cloze tests and the CELT 
and TOEFL subtests and the cocaposite proficiency score (see Table 1). Ab 
Table 1 shows, performance in the open-ended cloze test correlated moder- 
ately strongly with composite proficiency (.67 and .64 for the two pas-* 
sages) and with TOEFL Reading Comprehension (.73 and .61). Also, the 
teachers-made MC test correlated moderately highly with the composite 
proficiency score (.62 and .74). 
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Table 1 

Concurrent Validity Coefficients 



Conposite 

English Proficiency Subteat* Proficiency 

Cloze test CELT-S' CELT-L TOEFL-RC TOEFL-V TOEFL-A Score 



Open-ended 



acadeaic passage 


.39 


.51 


.73 


.42 


.51 


.67 


popular passage 


.40 


.43 


.61 


.42 


.52 


.64 


Interlingual MC 












.45 


academic passage 


.39 


.31 


.42 


.21 


.45 


popular passage 


.39 


.35 


.57 


.40 


.53 


.58 


Revised Interlingual NC 












.42 


academic passage 


.29 


.29 


.56 


.16 


.36 


popular passage 


.28 


.08 


.39 


.31 


.32 


.38 


Intralingual MC 










.42 


.49 


acadeaic passage 


.33 


.41 


.53 


.35 


popular passage 


.43 


.49 


.64 


.42 


.55 


.66 


Teacher-made MC 














academic passage 


.35 


.44 


.55 


.53 


.54 


.62 


popular passage 


.48 


.48 


.69 


.66 


.59 


.74 


^Subtests, listed in 


order. 


are CELT- 


•Strticture, 


CELT-Listening, 





TOEFL-Reading Comprehension, TOEFL-Vocabulary , and TOEFL-Writing Ability. 



In general, the validity coefficients were highest for the open-ended 
and teacher-made MC tests, and coefficients for the intralingual MC test 
were next highest. The lowest correlations were those involving the 
revised interlingual MC test; this may have been due partly to the fact 
that this test contsiined only 20 items, in contrast with the 50 items for 
each of the other tests. 

For the MC cloze tests, correlations with the criterion measures were 
generally higher for the popular passage than for the academic passage. 
This may have resulted from the fact that the text of the acadeaic passage 
uaa similar in nature to the text of the passages in the TOEFL Reading 
Comprehension subtest. 

It appears that the open-ended cloze test is a slightly more valid 
measure '.»f English proficiency (particularly reading comprehension) than 
are mul tiple-choice cloze tests when an academically oriented science 
passage is used. However, the intralingual and teacher-aade MC cloze 
tests appear to be a'c least as valid as the open-ended cloze test when the 
text involves science as a topic of popular interests 
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68. Schrader, W. S., & Pitcher, B. (1970). Interpreting performance of 
foreign law students on the Law School Admission Test and the 
Test of English as a Foreign Language (Statistical Rep. No. 
70-25). Princeton, NJ: Educational Testing Service. 



Purpose 

Thia study examined the degree to which foreign students' performance 
in a pre-law school orientation program and later performance in law 
school was predicted by (a) TOEFL, (b) the Law School Admission Test 
(LSAT), and the Michigan Test of English Language Proficiency (MTELP). 
The study also assessed the amount of change in scores on TOEFL and the 
LSAT during the orientation program. 



Wtethod 

The subjects were drawn from a group of 121 foreign stiKients enrolled 
in an eight-week summer orientation program designed to prepare them to 
enter one of 23 law schools in the United States. Scores v^re available 
for 112 subjects on all of the tests described below, and data indicating 
performance in the orientation program were available for 108 of these 
subjects. These 108 subjects came from 36 different countries, grouped as 
follows: Europe (N - 41), Asia (N - 33), Latin America (jJ - 24), and 
Africa (£ - 10). A total of 63 subjects had first-year average grades in 
one of four law schools: Columbia University, Harvard University, the 
University of Michigan, and New York University. 

The five-part TOEFL and the LSAT were administered by orientation 
program staff near the beginning and again near the end of the summer 
session. 

The LSAT produces a single score but has several parts: (a) Reading 
Comprehension requires the subject to read several passages and answer 
questions about them; (b) Reading Recall is similar but does not allow the 
subject to look back at the passages when answering the questions; (c) 
Data Interpretation requires the subject to answer questions about graphs 
or tables; (d) Principles and Cases measures reasoning similar to that 
required in legal studies; and (e) Figure Classification requires reason- 
ing involving geometric symbols. 

The MTELP, administered before the subjects came to the United 
States, also produces a single score and has three parts: (a) Grammar, 
which taps understanding of English grammatical structure; (b) Vocabulary, 
which tests knowledge of trords commonly encountered in university study; 
and (c) Reading Comprehension, which requires reading and answering 
questions about prose passages. {For a more complete description of the 
MTELP, see Summary No. I, Abadzi, 1976. J The Michigan Test of Aural 
Comprehension was also administered to the subjects, but the data from 
this test were not analyzed. 
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At the end of the orientation prograa the std>Jectt were graded In 
each of their two courses on • nine-point scale. Each instructor also 
rated the subjects on a four-point scale with regard to (a) their written 
English and (b) their spoken English. 

Average law school grades were obtained for each of 63 subjects. 
Because of possible differences in grading standards among schools, It was 
decided that the aean and standard deviation of law school grades and of 
all other measures should be asstncd to be mifonn for the four law school 
groups. Thus, score adjustnents were made such that each measure for a 
student was evaluated according to the student's score relative to other 
students In his or her own law school. 



Results and Conclusions 



Scores on the LSAT, total TOEFL, and each TOEFL srctAca except 
Reading Comprehension showed significant Increases over the eight-week 
orientation program. The mean initial TOEFL score of 570.1 was at the 
84th percentile of all '■'>relgn students seeking admission to U.S. colleges 
between February 1964 fld June 1969; the mean final score of 601.5 was at 
the 91st percentile. The mean initial and final LSAT scores, respectively, 
were at the 12th and 25th percentiles of all candidates given the LSAT in 
academic years 1966-67 and 1967-68. These gains may have been due to the 
orientation program or to such factors as living in the United States for 
eight weeks or the experience of having taken the test once before. 

Analyses involving prediction of orientation-program grades included 
the Initial LSAT, the MTELP, and initial TOEFL (including part sci s) 

. m ^ ^ ^ . m J ^^^^^ A<iratn4 r%AA A1 



(N - 108). First, correlations among predictors 
tliree (of 28) correlations were above .50. The LS 
TOEFL total and .52 with the MTELP; the latter t 
with each other. Among TOEFL part scores, the on 
highly with LSAT was Reading Comprehension (r - . 

.55). Correlations with the MTELP were in the lo^ ™ *w «^ 

TOEFL sections: English Structure, Vocabulary, knd Writing Ability; 
correlations with the other two sections were in jthe SOs. In general, 
there appears to be much overlap in abilities measure! by the LSAT and the 
English language tests, although tl»se abilities are Jot identical. There 
is evidently a strong commonality between TOEFL and t le MTELP. 



e examined. All but 
correlated .66 with 
tests correlated .87 
that correlated most 
; other £*s - .45 to 
80s for three of the 



related .41 with the 
.29 with the initial 



Average grade in the orientation program coi 
initial LSAT (N • 108). Average grade correlate 
TOEFL and .18 with the MTELP, showing that predifction of academic per- 
formance from these English language tests was relaftlvexy poor. The fact 
that the MTELP was used as a basis for selectlqn may have reduced its 
predictive validity. 
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Orientat Ion-program Instructors' ratings of writing ability correlat- 
ed .35 with the Initial LSAT, .34 with the Initial TOEFL, and ,19 with the 
MTELP; ratings o£ oral English correlated ,35, ,43, and ,33 with these 
three tests, respectively. The two course grades correlated ,57 with each 
other; the two ratings of writing ability correlated .59, and the two 
ratings of oral English correlated .62. 

Post-orientation program TOEFL scores correlated .36, ,42, and .48, 
respectively, with grades, written English ratings, and iral English 
ratings. These correlations were slightly higher than the correlations 
with Initial TOEFL scores. Correlations of LSAT with these three measures, 
respectively, were .37, ,34, and ,32. which were slightly lower than those 
with the initial LSAT. 

Prediction of grades in law school was based on 63 subjects [and the 
data were subjected to the above-mentioned adjustment] so the results must 
be Interpreted with caution. Of the scores on the initial tests, five 
scores correlated higher than .40 with grades: LSAT (.47), TOEFL (,41), 
MTELP (,42). and two TOEFL sections. Writing Ability (,47) and Reading 
Comprehension (,43), (r's for other TOEFL sectioas ranged from ,27 to 
.31.) 

The multiple R of Initial scores on the LSAT and TOEFL scores with 
law school grades was ,48, Given that LSAT and TOEFL Writing Ability each 
correlated .47 with grades, joint use of TOEFL and the LSAT does not 
appear to improve prediction compared with use of either one of these two 
tests alone. 
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69. Scoon, A. R. , & Blanchard, J. D. (1970, March). The relation of % 
test of English as a second language to measures of Intelli- 
ge nce, achievement, ard adjustment In a sample of American 
Indian students . Paper presented at the fourth annual TESOL 
Convention, San Francisco. (EKIC Document Reproduction Service 
No. ED 039 530) 



Purpose 



This study investigated performance on, and interrelationships among. 
Intelligence and pert-onality tests and TOEFL for a group of bilingual 
American Indian students. Several hypotheses were Investigated. The 
first hypothesis was that Indian students would manifest an English 
ability skill, as measured by TOEFL scores, that was discrlminable from 
personality adjustment and Intelligence test scores. A second hypothesis 
was that low TOEFL scores would be associated with low scores on the 
personality adjustmen test. A third hypothesis was that scores on a 
verbal intelligence test would be significantly different from ecores on a 
nonverbal Intelligence test for these Indian students. Further questions 
for research were (a) whether there would be sex differences on these 
tests and (b) whether these Indian students would show changes in their 
test scores from the ninth through the thirteenth year of school. 



Method 



The subjt ts were 1A2 bilingual American Indian students at the 
Institute of American Indian Arts in Santa Fe , New Mexico. Students 
attending this school were selected for admission on the basis of their 
artistic ability. The sample Included 78 male and 64 female subjects from 
11 Indian languflfe families. 

The Instruments used in the study were the five-part TOEFL, the Otis 
Quick Scoring Mental Ability Test, Gamma, which was regarded as a test of 
verbal intelligence (VIQ), the Chicago Non-Verbal Examination, regarded as 
a test of nonverbal intelligence (NVIQ), the Iowa Test of Educational 
Development (ITED), and the Bell Adjustment Inventory Revised (1962) 
Student Fo -m (Bell). iNo details about these tests are given. However, 
for a brief oescription of the ITED, see Summary No. 12, Blanchard and 
Reedy, 1970.) 

Not all subjects took all tests. Subjects In Set 1, which consisted 
of 78 students in grades 9-12, took the VIQ, NVIQ, TOEFL, ITED, and Bell 
tests. Set 2 subjects, consisting of 16 students in grades 9, 11, '•ud 12, 
took the VIQ, NVIQ. TOEFL, and ITED tests. Set 3 subjects, consisting of 
33 students in grade 13, took the VIQ, NVIQ, TOEFL, and BELL tests. Set 4 
was the combination of subjects in Sets 1, 2, and 3 plus six additional 
subjects; analyses for Set 4 involved only the VIQ, NVIQ, and TOEFL 
scores. 
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Results 



Over all subjects, NVIQ scores were found to be Mgher (mean « 107*7) 
than VIQ scores (mean - 90*6); this difference was statistically signifi- 
cant beyond the •001 level. The correlation between NVIQ scores and VIQ 
scores was .SZ* Some Indian language groups manifested higher NVIQ and 
VIQ scores than others* <rhe average ITED scores for these subjects were 
generally low — below the lOth percentile nationally on all subscales of 
the ITED, 

The' mean total TOEFL score of the subjects was 483. Scortes on the 
TOEFL sections shoved a steady increase from the ninth through the thir- 
teenth grade, with the exception of scores on the Listening Comprehension 
and English Structure sections, which showed a decline in the twelfth 
grade. Correlations betv^en TOEFL subscores and total score and among the 
ibtests of the ITED were modest to high; all of these correlations ranged 
crom .48 to .SB* The correlations across the TOEFL and ITED subtests were 
not as high, overall, as were the correlations among subscores and 
total TOEFL score » Correlations between TOEFL (part and total) scores and 
the scales on the Bell test were low, and none reached statistical signi-- 
f icance* 

Seventeen percent of the 133 subjects who took the TOEFL scored 550 
or above, indicating no restriction based on their English proficiency for 
college work* Fifty-three percent of the subjects scored in the range 
450-^549; these students were judged to need some classwork in English as a 
second language (ESL) but no restriction in their college course load. 
Twenty^-nlne percent of the subjects scored in the range 300^449; these 
students were judged as needing considerable ESL classwork and a reduced 
college class load. Only one subject scored in the range 200-299, indl-- 
eating the need for full-time ESL class work» 

Factor analyses were performed separately on the data for Sets 1 
through 4. Across all analyses, the TOFFL, intelligence, and achievement 
test scores emerged as Iropor-tant contributors to a first, general factor* 
This factor was interpreted as English language fjicility. 



Conclusionc 



TOEFL appears to be a valid measure of English language skills of 
Indian students. The observed variation in TOEFL scores was similar to 
that typically observed ^orrg foreign students. ITED scores were judged 
to measure language ability, although the ITED generally appeared too 
difficult for these subjects. Colleges might benefit from evaluating the 
TOEFL scores of Indian students in making admissions decisions^ as the 
ability of Indian students to succeed academically might be associated 
with their English proficiency. 
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70. Sharon. A. T. (1972). English proficiency, verbal aptitude, and 
foreign student success in American graduate schools. Eouca-" 
tional and Psychological Measurement , 32. 425-A31. Also printed 
as Test of English as a Foreign Languag e as a moderator of 
Gr aduate Record Examinations scores la the predic tion of foreign 
•S tudents' gradii in graduate school . 1971 CETS Research ^illetin 
No. 1l-50). Princeton, NJ: Educational Testing Service. (ERIC 
Document Reproduction Service No. ED 058 304) 



Purpose 



This study sought to investigate how TOEFi- would add to the predic- 
tive validity of the Graduate Record Examinations (GRE) Aptitude Test. 
One hypothesis was that TOEFL, as an English proficiency test, and the 
GRE, as a measure of academic aptitude, should yield different information 
about the preparation of foreign candidates for graduate school. A 
further hypothesis was that GRE verbal (GRE-V) scores should be better 
predictors of graduate school achievement for students with high TOEFL 
scores than for students with low TOEFL scores. 



Method 

A total of 975 foreign student? weri 'Studied. Scores on the five 
part TOEFL, GRE Aptitude Test scores, and graduate school grade-point 
average (GPA) were supplied by 24 graduate schools. [For a brief descrip- 
tion of the GRE Aptitude Test, see Summary No. 4, American Association 
.... I97li^ Test scores were presumably obtained via preadmissions test- 
lnR--e.g., International tes; administrations, in the case of TOEFL. J 
Information was also obtained regarding major field of study, numoer of 
semesters upon which the GPA was based, and whether the subjects had 
withdrawn from graduate school. For purposes of the study, major areas of 
stuUy were categorized as (a) engineering, technology, and mathematics, 
(b) natural sciences, and (c) other. 



Results and Conclusions 

The TOFFL and GRE scores of the 975 subjects in this study were 
noticeably different from those of reference samples of examinees who had 
taken TOEFL and the GRE. The average TOEFL score of the present sample 
was 537 as compared to an average score of 487 for 113»?75 foreign stu- 
dents who had taken the TOEFL in the period February 1964 through June 
1969. The average GRE scores of the students in the present study were 
348 for verbal (GRE-V) and 609 for quantitative (GRE-Q) score. These 
scores contrast with average scores of 516 for GRE-V and 524 for URE-Q for 
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those taking the test in the period May 1966 through April 1969. (The 
latter examinees, who numbered approximately 539,000, were almost all 
native Americans.) 

GRE-V score correlated .70 with TOEFL score for subjects in the 
present study. While this correlation is high, it ia nonetheless consis- 
tent with a hypothesis that the tests are not measuring exactly the same 
underlying abilities. x 

The relationships of CPA with GRE-V, GRE-Q, and TOEFL scores were 
investigate' separately for different major areas of study as well as for 
all majors combined. Linear regression analysis was used to determine how 
well CPA was predicted by combined GRE-V and TOEFL scores, and by combined 
GRE-Q and TOEFL scores* A special linear regression procedure was used 
that accommodated for differences in grade distributions across the 24 
schools that were studied. 

Over all majors, GRE-Q correlated .32 with GPA; this correlation was 
hi^.her than that between GRE-V and GPA (jr » .24) and that between TOEFL 
and GPA (£ « .26). Grade-point average correlated more highly with GRE-Q 
than with TOEFL for both engineering technology , and mathematics (GRE-Q: 
r « .39; TOEFL: £ - .21) and the 'natural sciences (GRE-Q: r » .59; 
TOEFL: £ - .39). For the "other" category, however, GPA coTrelated 
slightly higher with TOEFL (£ - .39) than with GRE-Q iv_ - .28) or with 
GRE-V (£ - .35). Combining TOEFL with GRE-V scores or TOEFL with GRE-Q 
scores was judged not to result in significantly higher predictability of 
GPA than was observed when the single best predictor was used, [No tests 
of statistical significance are cited to support thie statement.] 

Associations between GPA and GRE scores were examined based on 
whether the subjects had scored in a low, middle, or high TOEFL score 
range. These associations vrere investigated separately for (a) engineer- 
ing, technology and mathematics and (b) other. (The natural sciences area 
was not Investigated. because of the small number of subjects in this major 
area.) For subjects with TOEFL scores in the low and middle. ranges there 
were stronger relationships between GPA and both GRE-V and GRE-Q scores 
Chan had been observed when all levels of TOEFL scores were canblned. For 
subjects with high TOEFL scores, howeve-, there was no strong indication 
of a similar pattern of results. Further, prediction of GPA by GRE-V 
scores was better for subjects with low and middle TOEFL scores than for 
subjects with high TOEFL scores in the area of engineering, technology, 
and mathematics. This result was nor conslF-^ent with the hypothesis 
that GRE-V scores should correlate better with GPA among students with 
high TOEFL scores than among those with low scores. 
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71 Shay, H. R, (1975). Affect IsicJ of foreign students' language 
proticlency on academic perforiaance. (Doctoral dissertation, 
West Virginia University. 1975). Dissertation Abstract s Inter- 
national , 36 , 1983A-1984A. (University Microfilms No. 75-21, 
931) 



Purpose 



This study ascertained the degree to which academic success of 
foreign graduate students in a U.S. university is related to (a) Graduate 
Record Examinations (GRE) Aptitude Test scores and (b) TOEFL scores. The 
study also inquired whether the predictive relationships for the GRE tend 
to differ for foreign and American graduate students. 



Background 



The literature for American and foreign students shows that the GRE 
is not a consistently reliable predictor of academic performance in 
graduate school. One problem for foreign students is that ^"8^;^^^:/^;: 
guage preparation in certain foreign countries is not adequate. TOEFL was 
developed to assist in making admissions decisions for foreign students. 
The test has been shown to correlate to a moderately high degree with 
other tests of English proficiency and thus appears to have good concur- 
rent validity. However, relatively low correlations have been found 
between TOEFL and measures of later academic success. It nas ^^e" ^^8"^^ 
that TOEFL's primary function for graduate students is as f moderator 
variable, to be used along with the GRE and othe- data in making admis- 
sions decisions. 



Me chod 



The subjects were graduate students who attended West Virginia 
llnlversitv between May 1964 and May 1974. There were four groups of 
subrects:' 174 foreign students who completed 30 hours toward a graduate 
degree (Group 1); 61 foreign students who failed to complete degree 
requirements (Group 2); 145 American students who had been "^^^^g'^^f^^^^ 
at west Virginia University (Group 3); and 165 American students who had 
been undergraduates elsewhere in the United States. <Gr°uP^V «r!^v 
subjects in Groups 1 , 3 , and 4 were matched according to ^^^Id of study 
and year of graduation. The foreign students were drawn from 31 different 
countries. 

Graduate Record Examinations verbal (GRE-V) and quantitative (GRF-Q) 
Aptitude Test fccores were available from preadmissions testing for 166 
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students in Group 1, 14 in Group 2, 144 in Group 3, and 164 in Group 4. 
(For a brief description of the GRE Aptitude Test, see Summary No, 4, 
American Association 1971.] Scores on the five-part TOEFL were 

available from preadmissions testing (presumably International administra- 
tions) for 69 students in Group 1 and 55 students in Group 2. (Mean TOEFL 
scores of Groups 1 and 2 were 507 and 499, respectively; SDs = 52 and 
42J — 

Grade-point averages (GPAs) were calculated for all students at three 
stages in their graduate training: after completion of (a) 9 credit 
hours, (b) 21 credit hours, and (c) 30 credit hours. 



Analyses of variance showed no significant differences in GRK scores 
between groups of foreign students (Groups 1 and 2) or between groups of 
American students (3 and 4). However, significant differences in both the 
GKE-V and GRE-Q scores were found between the principal group of foreign 
students (Group 1) and each of the two groups of American students (3 and 



Correlational analyses performed separately for 9-, 21*-, and 30-cred*- 
it hour GPA revealed the following effects: GRE-V did not correlate 
significantly with any of the GPAs for Group 1 (£*s ^ .06 to .09) or Group 
3 (all £'s ^ .15) but showed low significant correlations with the GPAs 
for Group 4 (£*s ^ .18 to .29). GRE-Q showed low but signific ; ' correla- 
tions with the GPAs for Group 1 (.26 to .27), Group 3 (.25 . .28), and 
Group 4 (.16 to .25). Correlations for Group 2 were based on too few 
cases to Interpret (^J ^ 14). These data show that GRE-V failed to predict 
academic success for most groups and, while GRE-Q generally correlated 
significantly with GPA, this score accounted for only a small percentage 
of the variance in students' GPAs and thus may be of limited value as a 
predictor of academic success. 

For the principal group of foreign students (Group 1), GRE-V was 
significantly correlated with total TOEFL score ( r ^ • 52) and wi th scores 
on four of the TOEFL subtests: Listening Comprehension (.30), Vocabulary 
(.46), Reading Comprehension (.43), and Writing Ability (.41)* Thus, 
there appears to be at least some commonality in skills measured by these 
two testst 

For the principal group of foreign students (Group 1), correlations 
were computed between TOEFL and GPAs based on 9-, 21-, and 30-credit 
hours. Correlations betw.en total TOEFL score and the GPAs were very low 
(ranging from .08 to .12). Of correlations involving the TOEFL subtests 
and GPAs, the only significant ones were those between English Structure 
and (a) 21--credit GPA (r » .29) and (b) SO'-credit GPA (£ « .33). All 
other correlations fell between --.10 and +.20 and were nonsignificant. 



Resvilts and Conclusions 
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These low TOEFL-GPA correlations indicate that TOEFL is not a reliable 
predictor of academic success in graduate school. 

It is recommended, among other suggestions, that admisrions decisions 
be based on undergraduate transcripts and that currently used verbal 
aptitude and language proficiency tests be excluded as admissions criter- 
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72. Sokari, H. (1981). Predictors of college success among foreign 
students from various ethnocultural backgrounds (Doctoral 
dissertation, University of San Francisco, 1980). Dissertation 
Abstracts International , 41, 35A3A-35A4A. (University Mlcro- 
fllms No. 81036S6) 



Purpose 



This study examined the role of several variables In predicting 
college grade-point average (GPA) for foreign students In two colleges. 
Predictor variables Included TOEFL scores, high school grades, sex, age, 
resident status (resident vs. connauter), college major, scores on the 
College Board Scholastic Aptitude Test (SAT), number of college imlts 
(credits) completed, and school attended* 



Background 



Three studies reviewed show that TOEFL appears to be a moderately 
good predictor of college success. However, these studies were limited in 
that two of them focused only on Asian students, and these studies did not 
compare TOEFL with other predictors. Research suggests that potential 
predictors Include sex, age, high 8clu>ol grades, college entrance examina- 
tion scores, resident status, and college major. 



Method 



The subjects were 420 tmdergraduate foreign students at two Catholic 
colleges in northern California (207 subj'^cts from College 1, and 213 from 
College 2), each of which enrolls approximately 3,500 to 4,000 students 
annually. [The specific colleges are not named.] The total sample 
included 196 males and 224 females, drawn from 35 different countries of 
origin. The subjects ranged In age from 18 to 32 years, and TOEFL scores 
ranged from 412 to 660. 

TOEFL scores were available from school records for College 1 only 
{these are presumably scores on the three-part TOEFL obtained via Inter- 
national or Special Center administrations]. Data for other predictor 
variables were available for both schools (except that resident status ma 
not a variable for College 1, as all students there were comsmiters). Data 
available from school records included (a) scores on the SAT [for a brief 
description of the SAT, see Summary No. 2, Alderman, 1982] ; (b) average 
high school grade [whether transformed to a single scale is not Indicat- 
ed]; (c) age of the subject; (d) resident status (i.e., resident on carapus 
vs. commuter); (e) major field; (f) ntaber of scholastic units cmspleted; 
and (g) ethnocultural backround, grouped Into seven categories: Africa 
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(Gaobia only). Oceania, (Cuba and Guam), North America (Canada), Europe 
(10 couPtries), Near and Middle East (seven countries). Middle and South 
America (six countries), and Fat East (eight countries). 



Uesults 



Stepwise multiple regression an^yses were performed separately for 
College 1 and College 2, with CPA as the criterion and each of the other 
variables mentioned above as predictors. The analyses Included three 
scores from the SAT: verbal (SAT-V), mathematics (SAT-M), and total 
(SAT-V + SAT-M) scores. For College 1 (the college for which TOEFL scores 
were available), SAT-M score accounted for most of the total variance (57 
percent) and, after the effect of SAT-M had been taken out, other vari- 
ables accounted for «-gliglble amounts of variance (each 4 percent or 
less). For College 2, average high school grade accounted for 26 percent 
of the total variance in GPA, SAT-M score added about 9 percent, and sex 
added 6 percent; other variables accounted for negligible amounts of 
variance (i.e., each 4 percent or less). Analysis for both colleges 
combined showed that SAT-M accounted for the greatest amount of variance 
in GPA (28 percent), with sex accounting for an additional 8 percent, 
and average high school grade an additional 6 percent; other variables 
accounted for negligible amounts of variance (3 percent or less). 

A stepwise multiple regression analysis was also performed for 
College 1 with TOEFL as a criterion and all other variables. Including 
college GPA. as predictors. In this analysis, total SAT^^accounted for the 
largest amount of variance in the TOEFL score (27 percent), with other 
variables accounting for negligible amounts of variance (each 1 percesmt or 
less). 

Additional analyses were also perfonaed to test the effects of each 
predictor separately on college GPA. Each analysis consisted of a contrast 
between groups, with the mean difference in GPA compared to the cotmon 
standard deviation. These analyses showed significant differences in GPA 
as a function of college attended (grades at College 1 were higher than 
those at College 2) and sex (males outperfonaed females), with the level 
of significance greater for students from male-dominated geographic 
regions, especially for certain college majors (humanities, physical and 
life sciences, and business administration). GPA was also significantly 
related to the subject's age (older subjects outperformed younger stu- 
dents) and to average high school grade for College 2. Also related to 
GPA were SAT scores, particularly SAT-M, and number of scholastic units 
completed. There was a suggested relation between resident status and 
GPA, with residents performing better than commuters, although the lack of 
resident students at one college prevents strong conclusions in this 
regard. Finally, the relation of TOEFL to GPA was examined for subjects 
at College 1 by dividing these subjects into two groups according to their 
TOEFL scores: 412 to 609 and 610 to 660. No significant difference 
between groups was found, indicating a failure of the TOEFL score to 
predict college GPA. 
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Concluslons 



The results luggest that TOEFL is not a good predictor of college 
academic perforaanca. Of the variables studied here, the beat predictor 
appeared to be the SAT~M score. Also of soae value as a predictor Is the 
student's sex (particularly for students froo male-doolnated societies) 
and, to a lesser degree, the student's age* The results regarding effects 
of average high school grade are mixed, as a significant relation of this 
variable to CPA was found for one college but not the other. Perhaps the 
cole of high school grades is moderated by the student's major and/or 
region of origin; the college in which a significant relation was found 
had a high percentage of Middle and South Americans majoring in engineer- 
ing, whereas in the other college, more than half the foreign atudents 
were froo the Far East and there was a higher percentage of business 
administration majors. 
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73. St. Martin. G. M. (1979). Effect of solourner hou sing altuatlona on 
second language acquisition . Paper presented at the meeting of 
tiie Society for Intercuxtural Training, Education, and Research, 
Mexico City. (ERIC Docunent Reproduction Service No. ED 183 

005) 



Purpose 



This study sought to detenaine whether living with an American faoily 
while taking an intensive English language course would Improve foreign 
students* learning of English. 



Method 



The subjects were 83 students enrolled in a 14-week Intensive^Engllsh 
language course who had chosen to live with American families ("hoaestay 
subjects") and 83 other students ( "non-home stay subjects") who were 
matched with them in scores on the Michigan A or Michigan Placement Test. 
iThe structures of these tests are not described.) In most cases, pairs 
were also matched for sex and native language. Non-hooestay subjects 
lived in dormitories or apartments, typically with, or close to, speakers 
of their native languages. Eleven different native languages were repre- 
sented. 

Instruction consisted of 22.5 total hours per week of classes In five 
areas: spoken English, grammar, composition, reading, f'**.}?^;^*^;'^- *J 
the end of the U-week term, all students were graded all areas «cept 
the laboratory. Also, most subjects took the three-part TOEFL Ipre.unably 
at International or Special Center administrations! , yielding 69 pairs of 
subjects on which analyses involving TOEFL were based. 



Results and Conclusions 



Analysis of variance showed significant differences between hoaestay 
and non-homestay subjects in total TOEFL score and in two subsections: 
(a) Listening Comprehension, and (b) Reading Comprehension and Vocabulary; 
the difference for the third section. Structure and Written Expresaion 
section, was In the same direction but was not significant. Analyse* of 
variance also showed significant differences in favor of hoaaitay subjacta 
in grades in all areas: spoken English, grammar, reading, and coaposl- 
tlon. 

The results support the assumption that the best way to acquira 
language competence is to live lo an environment In which It Is used. 
Aaiong possible reasons for this is that homestay students may be encour- 
aged to experiment with English in a relatively relaxed atmosphere. The 
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preient results, hov«ver, night also be due to the f«ct that sttidents who 
choose the hoaestay sltuotlon are less shy and have a more positive 
attitude toward English and English language speakers than do students who 
do nut choose this situation. 
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74, Stevenson, D, K, (1975), A prellainary Investigation of construct 
validity and the Test of English as a Foreign Language (Doctoral 
dissertation, University of New Mexico, 1974), Dissertation 
Abstrac ts International , 36, 1352A. (University Microfllnis No. 
75-ld. 664) 



Purpose 

The first portion of this thesis discusses the assessment of validity 
in relation to TOEFL and language tests in general; the second portion 
describes a project to develop an oral cloze test and determine its 
relationship to TOEFL and other measures. 

Background 

Several forces led to development of TOEFL, including the need for a 
language measure to use in selecting foreign students for admission to 
colleges and universities In the United States and Canada. Although 
TOEFL appears not to be a suitable predictor of college performance, it is 
most important that TOEFL assess the language proficiency required for 
academic success. A problem is that it has been difficult to define such 
a requirement. Yet the validity of TOEFL as a measure of English language 
proficiency needs to be established. 

Content validation is one approach. Unfortunately, linguistic theory 
does not iriclude specifications that can provide a firm basis for content 
validation of TOEFL, Expert judges can only determine if the content 
seems reasonably in accord with theory. Even then, the content selected 
by such judgmental procedures might prove to relate more to a foreign- 
speaker standard than to the more appropriate native-speaker standard. 

A second method is criterion-related validation — determining if the 
test is sufficiently related to another, accepted measure of English 
proficiercy, A problem, however, is that it is difficult to find a 
criterion measure whose own validity has been clearly established. Also, 
if the criterion is to be a behavior sample, appropriate quantification 
poses some difficulty, 

A third approach — construct velidation — is advocated here. This 
approach involves demonstrating (a) that two or more intended indices of 
the same skill nr** relatively highly related (thus showing "convergent 
validity**) an' hat two intended indices of different skills are less 

highly relat -s showing "discriminant validity"). The discriminant 

validity of th* TOEFL subtests nay not be sufficient to conclude that 

these different parts are measuring different skills (cf,, data from Pike, 
1979), 
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A aultltralt aultioethod approach to examining the validity of the 
TOEFL Listening and Reading Comprehension section is advocated. ^ By 
this method, convergent and discriminant validity would be denonstrated if 
correlations among tests of a single skill, using different methods, were 
higher than correlations hetmen tests of different skills, using similar 
methods. In this regard, it is proposed that the TOEFL Listening and 
Reading Comprehension subtests be administered along with an oral and a 
inrltten cloze test in order to examine interrelationships among these 
tests. The research summarized below Is an effort to develop an oral 
cloze test that might be used for this purpose and to determine its 
relation to TOEFL and to another test of listening ability. 



Method 



[Development of the oral cloze test Included several phases, and only 
the phase involving comparison with TOEFL Is summarized here. Development 
of a written cloze test, which was not compared with TOEFL, is also not 
discussed. ] 

The subjects were 101 foreign students at Indiana University, 
46 of vhou were included in the analyses involving TOEFL. 

In the oral cloze test, the subjects listened to two passages total- 
ing 434 words of difficulty appropriate for junior-high level native 
English speakers. Every seventh word waS replaced '^th a bell sound, 
and a six-second pause was inserted after every sentence to facilitate 
responding. The subjects guessed each deleted %iord by writing it on a 
separate answer sheet. Each passage was repeated fotxr times. Responses 
were scored by an exact-word criterion (I.e., only the precise word that 
had been deleted was scored correct) . 

A multiple-choice version of the noise test (see Gr adman & Spolsky, 
1974) was also administered. In this test, the subjects listened to 50 
sentences with a whlie noise overlay and, for each sentence, chose the one 
sentence out of five listed on an answer sheet that matched tte sentence 
heard. Scores on the five-part TOEFL [presunably obtained at Internation- 
al administrations J were available from the students' institutional 
records. 



Results and Conclusions 



For the full sample of 101 students, the oral cloze tept showed a 
suitable score range (0 to 36, of a possible 50) and KR-21 reliability of 



Xampbell, D. T. , & Flske, D. W. (1959). Convergent and discriminant 
validation by the multitrait-mtdti method matrix. Psychological Bulletin , 
56, 81-105. 
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.84. For the 46 students for whom TOEFL data were available, the score 
range was 2 to 36, and the KR-21 reliability was .76. For the noise test, 
a KR-21 reliability of .51 was observed for these 46 students. 

Correlations were computed among the oral cloze test, the noise test, 
and the five TOEFL subtests. Initial results of interest included the 
following: (a) Intercorrelations among the TOEFL subtests were not as 
high as in most studies (here, .05 to .67, median - .54 ); (b) the oral 
cloze test correlated most highly with the noise test (.67); and (c) 
correlations with the TOEFL subtests were higher for the oral cloze test 
(.29 to .51) than for the noise test (.19 to .45). Of particular interest 
is that the oral cloze test correlated more highly with TOEFL Reading 
Comprehension (.51) than with Listening Comprehension (.32). This last 
result implies either (a) that the oral cloze test may not be an effective 
test of listening ability, or (b) that the TOEFL subtests may be deficient 
in construct validity. 

A factor analysis of the seven variables was performed, and loadings 
on two factors were generated by varimax rotation. The highest loadings 
on one factor involved the oral cloze test, the TOEFL Listening Comprehen- 
sion subtest, and the noise test; the highest loadings on the other factor 
involved the other four TOEFL subtests. Although caution is warranted in 
interpreting the data from such a small sample, one factor observed here 
is tentatively identified as a listening factor and the other as a general 
language proficiency factor. 

The oral cloze test, developed as an experimental meastare, shows 
promise as an instrimient for further research on the construct validity of 
TOEFL and other language measures. 
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75. Stover, A. D, (1982). Effects of language admission criteria on 
academic performance of non-native English- speaking stiKientr 
(Doctoral dissertation, University of Arizona, 1981). Disserta- 
tion Abstracts International , 42 , 4374A-4375A. (University 
Microfilms No. DA 8207017) 



Purpose 

This study investigated the contribution of TOEFL scores, average 
grade in the final semester of pre-university classes in English as a 
second language (ESL), native language, and major area of study to predic- 
tion of first-semester university grade-point average (GPA). Two indepen- 
dent samples of students were employed in order to ascertain the gener- 
ali/;ability of results. 



Method 



The main sample of subjects were 159 students (98 undergraduate and 
61 graduates) who enrolled in the Center for English as a Second Language 
(CESL) at the University of Arizona between spring 1978 and fall 1979 and 
later enrolled in the regular academic curricultnn at the university. 
Admission to the University of Arizona required a TOEFL score of 450 or 
above and a minimum average grade of 2.5 in CESL classwork. These subjects 
had taken the three-part TOEFL. 

The validation sample consisted of 142 students (91 mdergraduates 
and 51 graduates) who enrolled in the CESL during summer 1976 to fall 
197 7, then enrolled in the regular academic curricidum at the University 
of Arizona. There was no minimum CESL grade requirement for university 
admission in force for the validation sample, which had matriculated 
earlier than the main sample. The validation sample had taken the five- 
part TOEFL. For both samples, TOEFL scores [presumably obtained in 
International administrations] and first-semester university GPA were 
obtained from school records. 

The predictive validity of TOEFL and the average CESL grade were 
examined for Arabic, Japanese, Spanish, and "other" language group*'. 
Also, the data were examined separately for several major areas of study; 
agriculture, business, engineering, liberal arts, fine arts, social 
sciences, and "others." 

Results and Conclusions 

In both samples, subjects with a minimum grade of 2.50 in the final 
semester of the CESL obtained an average GPA in their first semester of 
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reguXar academic study that was higher than tte minlraum CPA required to 
avoid academic probation; this finding applied for subjects who scored at 
or above 450 on TOEFL. The (three-part) TOEFL score of the main sample 
and the (five-part) TOEFL score of the validation sample were found to 
correlate .21 (£ < .05) and .16 ina) , respectively, with undergraduate 
CPA, TOEFL score did not correl'ine significantly with graduate CPA for 
either sample. 

There* did not appear to be any systematic differences in CPA among 
language groups or among major areas of study, either at the undergrs'* jcce 
or thf' graduate level. The Arabic language subgroup, however, obtained a 
lower mean TOEFL score than did the otaer language groups in the main 
sample of subjects. ^Regression analyses indicated that the graduate 
versus undergraduate variable was the strongest predictor of first- 
semester college; CPA and that the evcrage CESL grade was a str >nger 
predictor of first-semester college CPA than was the TOEFL score in both 
the main and val*dation samples. This result was not unexpcicted, given 
that ESL course >nce was similar to regular course experience at the 

university. Tl . session analyses also showed that pre-!lction of 
first-semester GP,^ . i TOEFL was better for subjects given the three-part 
TOEFL than for those giveu the five-part TOEFL. These analyses also 
indicated that language group was a significant predictor of university 
GPA for the validation sample but not for the m&in sanple of subjects. 

Graduate school f irst-f»femeet«t G I was not i»lgnlf Icantly relR'ced co 
an/ predictor variai.le except major irea. Major ar^a counted for 29 
percent of the varl>»ace o' first-se^e ter graduate GPA, at> higher grades 
»iere earnfcd by subjacts in the lib'-^r. 1 arts, ':lne wta, and social sci- 
ences than in other majors. 

A combination of TOEFL sco^-e and average CESL grauo. se^rt' effective 
for prediction of undergraduate college GPA. Scorcu on TOEFL and the 
average CESL grade appeared to m«asure different language related pheno«- 
ena. More reb«5arch is needed regarding the Enj,ii8H language requirements 
of academic work in college. A discrepancy between the main and validation 
samples in the importance of language group in determlnit»f GPA suggests 
that more attention needs to be given to how native language may influence 
English proficiency and academic achievement. 
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7h Swlnton, S. S, , & Powers, D. E. (1980). Factor analysis of the Test 
of English as a Forciign Language for several language groms 
(TOEFL Research Rep. No. 6; ETS Research Rep. No. 80-32). 
Princeton, NJ: Educational Testing Service. (ERIC Document 
Reproduction Service No. ED 218 921) 



Purpose 



TMb 8(;udy sought to Identify the component abilities fhat are 
measured by TOEFL and to determine whether there are differences aaong 
language groups In the patt rns of abilities tested. Toward this end, 
factor analysis was used to determine the factor structure of the test for 
each of seven major language groups. 



Met^d 



Approximately 600 to 1,000 subjects from each of seven language 
groups w.?.re chosen frcnn the total population of examinees at the November 
1976 International TOEFL administration. The langtiage groups were Arabic, 
Chinese (non-Taiwanese;, Farsi, Japanese, and Spanish, along with group- 
ings of African and Gr.cmaniw languages. Represented in th^:: African group 
were Yoruba, Ibo, Eiik, and other West African languages; and in the 
Germanic group, German, Dutch, Swedish, Norwegian, Danish, and Icelandic. 

Each item in the three-part TOEFL (excluding one item omitted from 
operational scoring) was used as a variable in the analyses. For each 
language i"o-.p, the" matrix of tetrachoric Interitem correlations was 
computed. (Xhis cb-r '-Station required that items ar.swered correctly 
by more than 96 percent of a group be omitted.) Factor analysis was 
performed, and a four-ractor varimax rotation was applied for each lan- 
guage group. An additional factor analysis was also performed using an 
orthogonal rotation to fit the design structure of the test, as described 
below. 



Results 



There was subtantial variation among language groups in mean TOEFL 
scores, with the Germanic group receiving the highest scores and the Far si 
speakers the lowest. Also, the patterns of subtest scores differed 
somewhat across language groups; speakers of African languages, for 
example, ranked second on Structure and Written Expression but sixth on 
Listening Comprehension. 

Preliminary data from the factor analysis suggested that three to 
five factors are appropriate to account' for the performance of each 
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language group except the Germanic group, for which as many as eight 
factors might be appropriate. The four-factor varlmax rotation was 
applied and, for each language group, the nuaber of Itons with the highest 
loadings on each factor was determined. Listening Comprehension items 
were found to form a separate factor for all language groups. Of the 
Items in the Structure and Written Expression section, the wltten expres- 
sion items clustered with the structure items for some native languages 
but not for others, and the vocabulary items formed a dimension separate 
from the reading comprehension items in many groups. 

The four-factor solutions for each language group were then submitted 
to an orthogonal rotation to ensure that they would fit the d<%sign struc- 
ture of the test. The first factor was specified as that on which a 
sample of 14 listening comprehension items shoved maximum loadings; the 
second target factor was defined by maximum loadings of all 14 structure 
items; and the third target factor was defined by maximum loadings of a 
sample of 14 vocabulary items. The fourth factor remained unspecified. 

A clear, unambiguous factor relating to listening comprehension was 
identified for all language groups. Other factors, however, were subject 
to group differences. For the African, Arabic, Chinese, and Japanese 
groups, the majority of structure, written expression, and reading compre- 
hension items showed their highest loadings on Factoi II, while the 
majority of vocabulf»*y items showed their highest loadings on Factor III. 
For the Spanish and Germanic groups, on the other hand, the preponderance 
of structure items and written expression items showed their highest 
loadings on Factor II, while most vocabulary items and reading c<MBprchen- 
slon items showed highest loadings on Factor III. For Farsi speakers, 
these two factors were much less differentiated. For most language 
groups, there was no obvious association between Factor IV and any parti- 
cular item type. 

Correlations were computed between each factor and the subjects* sex, 
age, academic status, and the number of times TOEFL had been taken. The 
correlations chat were most consistent across language groups were those 
between Factor III and (a) age and (b) academic status, suggesting that 
vocabulary Items may be more susceptible than any of the other item types 
to training or experience. 



Conclusions 



Three major factors appear to underlie performance on TOEFL. The 
first Involves listening comprehension for all language groups. T^ie 
nature of the second and third factors depends on the language group in 
question. For the two Indo-European language groups, whose languages are 
most similar to English (the Germanic and Spanish groups), the second and 
third factors correspond with the current format of TOEFL, in which 
structure and written expression items contribute one subscore and reading 
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comprehenslon «ncl vocabulary Items contribute another. For most other 
groups studied — ^African, Arabic, Chinese and Japanese—reading comprehen- 
sion Iteas cluster with structure and written expression items, with 
vocabulary fozmlng a separate dimension* 

These results indicate the value of examining not only the total 
TOEFL score in assessing language competence but section scores as well. 
The results also suggest that, although the score on the third section of 
TOEFL is based on a combination of performance on reading comprehension 
and vocabulary it cats, there can be value in examining performance on these 
two item types separately, since for some language groups these item types 
apparently tap different factors* 

An important issue of recent debate concerns the degree to which 
language proficiency is divisible into separate language skills. The 
present results suggest that the answer to this issxie may depend on the 
sample used, as the Farsi speakers showed relatively little differentia- 
tion among factors, whereas the Germanic speakers stowed considerable 
differentiation, with other groups falling between these extremes. That 
these two language groups also represented the extremes in average TOEFL 
scores suggests that tl» degree of differentiation may relate to overall 
level of English proficiency. 
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77. Warner, C. J. (1982). A study of the relationships between TOEFL 



scores and selected performance characteristics of Arabic 
students In an individualised conpetency based training program 
(Doctoral dissertation. University of Tennessee, Knoxvllle, 
1981). Dissertation Abstracts International , 42 , A810A. 
(University Microfilms No. DA 8209009) 



This study addressed the following questions regarding Arabic train- 
ees in a special tvo-year vocational education program in the United 
States: (a) How does the distribution of TOEFL scores compare with the 
distribution of instrtKitors* performance ratings? (b) Is there a differ- 
ence in distribution of performance ratings given by English language 
instructors and ttose given by instructors in individual areas of special- 
isation? (c) How do TOEFL scores compare with students* class rankings? 
and (d) Do trainees in different job categories differ in TOEFL scores? 



Grade-point average (CPA) for foreign students may be affected by 
their relatively low English proficiency and instructors* attitudes 
toward them, thus obscuring the students* academic accoopllstaaents. The 
low relationship between TOEFL and academic success found in some studies 
may relate to this facr. and to the fact that TOEFL measures knowledge of 
formal English structure but not necessarily ability to coomunicate. A 
stronger relationship might be observed if, as in the present study, 
academic success were measured by instructors' ratings rather than CPA. 



Eighty-one individuals from [an unnamed J Middle Eastern country 
participated in a two-year training program in the United States. The 
program prepared trainees, both professional and technical, to participate 
In the development of vocational education centers in their home coun- 
tries. The subjects attended English-language training programs during 
the first six months and professional/ technical training classes for the 
remainder of the two years. English training was provided at six U.S. 
universities, and professional/ technical training was provided at four 
U.S. universities. 

Scores on TOEFL {presumably the three-part version] were obtained 
for all 81 subjects at the end of their English training; except where 
noted, these are the TOEFL scores used in analyses reported below. TOEFL 
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had al&o been administered to 61 of thp-^e subjects at the beginning of 
their English language training to pe' .^t assesfioaent of gain* Both sets 
of TOEFL scores were obtained froc U.S* Department of Labor records {the 
scores were presumably earned in International or Special Center adminis- 
trations] • In addition, subjects were rated on a five-point scale by 
their English language Instriictors and prof essional/ technical instructors 
on four variables; (a^ xademic ability, (b) dependability, (c) profes- 
sional attitude, and ability to comsminicate. 



Results and Conclusions 



The overall mean TOEFL score after English language training was 
373*1, (SD « 39.0} • The mean change in TOEFL score, for the 61 subjects 
4.^ at both times, was 34.6 points (SD « 38«4). 

The subjects %7ere placed into five groups based on their TOEFL 
scores. The middle group was defined by adding one-half SD and subtract- 
ing one-ha^f SD from the mean, and score ranges for groups 2 and 4 were 
equal to one £D* Hence, the TOEFL scores for the five groups were (a) 
less than 314,6, (b) 314.6 to 353.5, (c) 353.6 to 392.5, (d) 392.6 to 
431.5, and (e) greater than 431.5. The nmbers of subjects falling into 
these five groups — 1, 23, 40, 12, and 5, respectively — comprised the 
distribution of TOEFL scores. • 

The numbers of subjects falling into each of five acadefiic***perform- 
ance groups (poor, below average, average, above average, and outstand** 
ing) , as rated by their English language instructors, were 5, 19, 31, 21, 
and 5, respectively. A chi-square goodness^ of fit analysis showed that 
this distribution was significantly different from the distribution of 
TOEFL scores. Analyses were also performed involving distributions 
of English instructors* ratings on each of the other three variables 
described above and professional/ technical instructors* ratings on each of 
the four variables. All of these distributions differed significantly 
from the TOEFL distribution. (These results do not permit conclusions 
about the degree of relationship between TOEFL scores and instructors* 
ratings. Since scores were apparently provided to the investigator 
without Identification by subject name, it was not possible to conduct the 
types of analyses, such as correlational an^'ysls, that could indicate the 
degree of match between individuals* scores.] 

The distribution of ratings by English language instructors was 
compared with the distribution of ratings by professional/technical 
instructors for each of the four factors. For the academic ability 
factor, these distributions differed significantly, with the first group 
giving a modal rating of average** and the second group, "* above average.** 
The distributions also differed significantly for tYm professional atti** 
tude and ability to communicate factors, but not for dependability. On 
all factors except dependability, then the students were rated more 
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favorably by their profeasional/technlcal Instructors than by their 
English language Instructors. These results may have to do with differ- 
ences In frame of reference, as the English language Instructors' primary 
experience Is with foreign students preparing to enter college-degree 
programs, whereas the professional/ technical Instructors are accustomed to 
students stttdylng vocational education. 

The professional/ technical Instructors were asked to place their 
students into three groups: top third, middle third, and bottom third of 
the class. No significant difference in TOEFL scores among these groups 
was observed, as the three groups had mean TOEFL scores of 388.5, 362.9, 
and 357.1, respectively. 

Five subgroups of trainees were Identified: (a) administrator, 
coordinator (N - 13); (b) Instructor-educator, TV director, media produc- 
er/supervisor (N - 13); (c) curriculum developer, media developer, engi- 
neering technician, computer technician ( N - 7); (d) Instructor, main- 
tenance engineer (N - 39); and (e) media operator, technician, recording 
engineer, illustrator/artist (N - 9). Subjects were placed into three 
groups based on their post-training TOEFL scores; (a) below 353.6, (b) 
between 353.6 and 392.6, and (c) above 392.6. A chi-square test of 
Independent samples showed a significant relation between job classifica- 
tion and TOEFL score. Mean TOEFL scores of tte five groups were 379.2, 
382,2, 401.0, 366.8, and 357.0, respectively. (No significant relation 
was observ d between job classification and £aln in TOEFL score due to 
English training, however.) English proficiency thus seems to have been 
involved to some extent in selection of trainees for their jobs. 
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78. Wilcox. L. 0. (1975). The prediction of acada&ic success of under- 
graduate foreign students. (Doctoral dlssertatiout University 
of Minnesota^ 1974). Dissertation Abstracts International, 35, 
608AB. (University Hicrofilas No. 75-12, 178) 



Purpose 



This study examined the predictive validity of prior academic record 
and standardized test scores for students from Hong Kong enrolled at a 
state university and for students frcnn Vietnam enrolled at several insti- 
tutions. 



Method 



The subjects in this study were (a) 99 students from Hong Kong in 
their freshman year at the University of Wisconsin — Madison between 1968 
and 1973 and (b) 84 Vietnamese students in their freshman year at 16 U.S. 
institutions during 1967 and 1968. The subjects from Hong Kong all 
received financial support from their families, while the subjects from 
Vietnam were financed by the Agency for International Development. The 
Hong Kong group had taken the Hong Kong Certificate of Education Examina- 
tion (HKCEE), which Is used to determine which students will be permitted 
to enroll in a local two-^year course of study leading to enrollment in a 
university. There are two comparable versions of the HCKEE, one In 
Chinese and one in English; data from the latter were used in this study. 
Students take examinations in five to nine std>Ject areas and, in each, are 
graded on an eight-point scale. For this study, the student's Bcore w,b 
the simi of his or her six best scores. 

The Vietnamese subjects had taken the Baccalaureate Second Part 
Examination (Bac II), which is required In South Vietnam for admission to 
a university and for deferral from the military draft. The Bac II con-* 
slsts of essay questions In philosophy, natural science, physics and 
chemistry, mathematics, and two languages; and imiltiple-ctu)ice questions 
in history, geography and civics. The results of each test are flighted 
and summed to provide a total score on a five-point scale. 

/ 1 subjects had graduated from secondary school in their home 
countries and had taken the flve**-pArt TOEFL (presimmbly in International 
administrations] and the College Board Scholastic Aptitude Test (SAT). 
[For a brief description of the SAT, see Sunmary No. 2, Alderman, 1982.] 
The subjects from Hong Kong had also taken at least two College Board 
Achievement (Ach) tests. College Board Achievement tests, offered in 
15 subject areas, consist of multiple-^cholce items testing students' 
knowledge of the particular stibjecta. The group from Rong Kong took all 
tests In their native country while the group from Vietnam was tested upon 
arrival in the United States. The Hong Kong sample was divided Into 
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Bubgroups by curriculum (liberal arts vs. engineering), year of enroll- 
ment, and sex. The Vietnamese sample was divided into those studying In a 
California university ar^d those studying at any of 12 other U.S. institu- 
tions. 

Test scores and prior academic record were correlated with first- 
semester grade-point average (CPA) for Hong Kong subjects, and with 
first-year CPA for Vietnamese subjects. Multiple regression analysis was 
then applied to the predictors to determine the best combination of 
variables for predicting CPA and the relative contribution of each vari- 
able. Both Bayeslan and least squares regression weights were applied and 
compared. Bayeslan procedures are considered preferable for analysing 
data from small samples. 



Results and Conclusions 



The correlation between the HKCEE and first-semester GPA for Hong 
Kong subjects was .40. This is slightly less than the correlation between 
high school performance and college GPA for U.S. students reported in the 
literature, although the correlation increased to ,50 when corrected for 
restriction of range. The correlation between the BAC II and first- 
semester GPA for Vietnamese subjects was .34; the correlation with full 
first-year GPA was .50. For this same group, high school grade average 
correlated .47 with first-year GPA. 

The correlations of these and other standardized tests with GPA are 
depicted in Table 1. The data show that SAT mathematics (SAT-M) scores 
were significantly related to GPA in both the Hong Kong and Vietnamese 
samples. These correlations are comparable to those commonly reported for 
U.S. students. For the Hong Kong group, the average score on three SAT 
achievement tests (ACH Ave) functioned as a useful predictor of GPA (r 
,49). When added to HKCEE scores, ACH Ave scores contributed flignlTi- 
cantly to the prediction possible from HKCEE alone. For both groups, the 
addition of SAT-M to the measure of high 8ctK)ol achievaaent (HKCEE or Bac 
II) increased the correlation with freshman GPA by about .10. 

TOEFL score was unrelated to GPA in the Hong Kong sample but corre- 
lated significantly with GPA in the Vietnamese sample. TOEFL did not 
improve the prediction of GPA when combined with other good predictors 
for either group. 

The two groups differed considerably in the mean and standard devia- 
tion of their TOEFL scores (Hong Kong: mean - 570, SD • 41; Vietnamese: 
aean - 459, SD - 78). Far greater variation in thc"Vletn«8ese than the 
Hong Kong auHJscts may partly account fpr the difference in correlation 
for the two groups. Also, a threshold variable may have been operating. 
That is, English skills and academic success may be related at low levels 
of proficiency but unrelated at levels above the threshold value. This 



ERIC 



218 



-211 



Table 1 

Simple end Multiple Correletlone of High School Acedeolc 
Achieveaent end Teet Scorea fd.th College CPA 



Group 


Predictor(e) 


Correlation with CPA* 


Hong Kong 


HKCEE 


• v 


N - 99 


SAT-V 


.01 




SAT-M 


,41* 




**ACH Ave 


.49* 




TOEFL 


.00 




HKCEE ACH Ave 


.56* 




HKCEE •¥ TOEFL ACH Ave 


.56* 


Vietnaaese 


BAC II 


.50* 


N - 84 


SAT-V 


.33* 




SAT-M 


.51* 




TOEFL 


.46* 




Bee II + SAT-^ 


.60* 




Bee II + SAT-M + TOEFL 


.60* 



Flrst-eeaester CPA for Hong Kong sample; first-year CPA for Vietnamese 
soaple. 

*p < .01 



might explain why the Hong Kong aubjecta showed no relationship between 
TOEFL and CPA while the Vietnamese aubjecta, whose English proficiency was 
lower, showed a aignificant relationship between these measures. 

The Hong Kong and Vietnamese samples irere subdivided according to 
year of enrollment, sex, and curriculum. None of the restating correla- 
tiona for a particular predictor or set of predictors was algnificantly 
different from the correlation for the total group. The least squares and 
the Bayesian regreasion procedures predicted eqtially well. 

A final analysis for the Vietnamese saaplc^amined whether English 
proficiency served as a moderator variable in merainlng the predictive 
validity of aptitude test scorea. This sample waa divided into high and 
low scoring groupa on TOEFL, with the midpoint being a acore of 450. 
No significant difference between groups was found in the correlations 
with CPA for SAT-V or for SAT-M, 
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79. Wilson, K. M. (1982). A comgaf tlve analyslB of TOEFL examinee 
characteriatlcs. 1977-1979 (TOEFL Research Rep. No. 11; ETS 
Research Rep. No. 82-27). Prlaceton, NJ: Educational Testing 
Service. 

Purpose 



To gain a better understanding of the role of TOEFL in the education- 
al plans of foreign students, this study examined the charactwristics and 
perforaance of foreign nationals who took TOEFL during the period froo 
September 1977 through August 1979 and reported that they were doing so 
in order to study in the United States or Canada. 



Method 



A history file was created using TOEFL program records containing 
data on personal and academic characteristics provided by examinees on the 
TOEFL answer sheet. These data Include th« exsmlnee's sex, age, native 
language, native country, country of residence, level of Intended degree 
program (and Intended department of stiidy for graduate level aspirants), 
previous TOEFL testing, and pattern of score reporting (i.e., designating 
or not designating institutions/agencies as score-report recipients). 
Descriptive statistics on these data were computed. Section and total 
scores on the three-part TOEFL were determined for subgroups on each 
variable. Finally, correlational analyses were performed in order to 
quantify the relationship between membership In relevant subgroups and 
performance on TOEFL. The analyses Included only International and 
Special Center testing program examinees who designated native countries 
and indicated that their reason for taking the TOEFL was to study at a 
university in the United States or Canada. A total of 235,738 examinees 
met these criteria. 



Results and Conclusions 

Of the many findings reported in this study, the following are some 
of the more salient. 

A total of 163 countries were named as native countries by two or 
more degree seekers. The 25 largest native country groups accounted for 
84 percent of ail degree seekers. Five countries accounted for 54 percent 
of all degree seekers; Asian and Mideastem countries accounted for 50 
percent and 23 percent, respectively. 

About half of all degree seekers were prospective undergraduate 
students, and about half were prospective graduate students. Almost 



218 



-213- 



one-third had taken TOEFL previously. More than sever in 10 veve male, 
and almost three In 10 were tested in the United States or Canada. The 
typical (median) undergraduate degree seeker was 20 years old, and the 
typical graduate degree seeker was 25. Only 50 percent designated insti- 
tutions to receive TOEFL score reports. Sixty percent of all prospective 
graduate students did not name specific deparlpents of study. Among the 
40 percent that did, 50 percent named one%»x the natuZi^l sciences, 20 
percent named a business school, 20 percent named one of tl e social 
sciences, and 8 percent named a subject In the hunanlties. 

Prospective graduate students performed better on TOEFL (mean total 
score 511) than did prospective undergraduates (499). Undergraduates 
typically outperformed graduate sttidents cn the Listening Comprehension 
section and graduate students typically scored higher on Structure and 
Written Expression and on Reading Comprehension and Vocabulary. Women 
tended to outperform men (mean scores "•513 and 502, respectively), and 
examinees tested in foreign centers did better than those tested domesti- 
cally (mean scores - 512 and 488, respectively). Repeaters tended to 
attain a lower mean score (496) than did those taking the test for the 
first time (505). The 57 pei'cent who did not request that official snore 
reports be sent to ins'iiitutions received a considerably lower mean score 
(486) than did the total group (505). 

For 129 native country subgroups with 15 or mc»-e degree planners, the 
following relationships were identified. The percentage of nonreportlng 
of scores to Institutions was Inversely related to TOEFL score (jr - -.55). 
The percentage of repeaters was Inversely related to TOEFL score (jr " 
-.64). Native cowitrles with a higher percentage of women examinees had a 
higher TOEFL score (£ - .40). On the average, examinees from developed 
countries included higher percentages of women, were younger, and had 
higher TOEFL scores (especially on the Listening Comprehension subtest) 
compared to examinees from developing countries. 

Based on the above findings, the following conclusions seem warrant- 
ed. Many examinees (perhaps the majority) do not apply to take TOEFL at 
or around an appropriate time of testing for admission for the following 
academic year. Those who designate institutions to receive score reports 
are closer to the time of enrollment than are those who do not. The 
latter group of examinees are primarily Interested in assessing their 
English proficiency to determine if they should apply for admission now or 
continue to study English and repeat TOEFL until they attain an acceptable 
score. 
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^0. Wilson, K, M. (1982), GMAT and GRE Aptitude Test performance in 
relation to primary language and scores on TOEFL (TOEFL Research 
Rep. No. 12; ETS Research Rep. No. 82-28). Princeton, NJ: 
Educational Testing Service. 



Purpose 



This study was conducted with two main goals: (a) to describe the 
performance of foreign examinees on the Graduate Management Admission Test 
(GMAT) and the Graduate Record Examinations (GRE) Aptitude Test in rela- 
tion to self-reported primary language (English vs. other) and in relation 
to the performance cf the total popidation examinees on these tests, 
and (b) to analyze the relationships betwe^ . performance on TOEFL and 
performance on the GMAT and GRE. 

Method 

Through the use of cross-file matching of data maintained by the 
GMAT. GRE, and TOEFL programs, a history file was constructed that con- 
tained data on the performance of examinees who took either the GMAT and 
TOEFL or the GRE and TOEFL between September 1977 and August 1979. [The 
GMAT is briefly described in Summary No. 64, Powers, 1980; the GRE Apti- 
tude Teat Is briefly described in Summary No. 4, American Association ...» 
1971; TOEFL scores were presumably obtained via International or Special 
Center administrations.] S^-parate analyses were conducted for foreign 
candidates who Indicated on their OiAT or GRE answer sheets that English 
was their primary language (EPL) or second language (ESL). EPL examinees 
were those who reported that they communicated better or were more fluent 
in English than in any other language. The subgroups %*ere further broken 
down according to self-reported citizenship status (U.S. vs. foreign). 

Th*» GMAT consists of verbal (GMAT-V) and quantitative (GMAT-Q) 
sptlti«l»; sections, and the GRE consists of verbal (GRE-V), quantitative 
(GRE-Q), and analytical (GRE-A) aptitude sections. For each group of 
examinees, means and standard deviations were obtained for section and 
total scores on each test. Also, a linear correlation was computed 
between the score for each section of the GMAT or GRE and the TOEFL 
score. 

Results and Conclusions 

GMAT/TOEFL 

Over 5,000 examinees took both the GMAT and TOEFL during the period 
in question. Table 1 shows mean scores on the two tests. Also indicated 
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are percentile ranks showing the standing of the exaniaees either (a) on 
the 6MAT relative to all examinees taking the (MkT betwt.en October 1977 
and July 1980, or (b) on TOEFL relative to all prospective graduate 
applicants taking TOEFL between September 1978 and August 1980. The 
typical GHAT/TOEFL examinee apparently was well above average in English 
language proficiency. The TOEFL maan (553) for this group was at the 77th 
percentile in the distribution of scores for all prospective graduate- 
level examinees. Percentile ranks on GMAT subscores were considerably 
lower for this select group of TOEFL examinees, except for the score on 
the quantitative section, which was above the mean for all examinees on 
the GMAT. While the foreign EPL examinees (from non-English-speaking 
countries) had higher verbal scores than the ESL examinees, Khey were 
still substantially below the mean for all GMAT examinees. 



Table 1 

Heans and Percentile Ranks on TOEFL and GMAT 
for TOEFL/GHAT Examinees 





Foreign EPL 


Foreign ESL 


EPL 


& ESL 




( N - 


1197) 


(N - 


3918) 


( N m 


5115) 




Mean 


Rank 


Mean 


Rank 


Mean 


Rank 


GMAT-Verbsi 


20.2 


25 


15.7 


13 


16.8 


18 


GMAT-Quan 1 1 1 a t ive 


27.5 


52 


29.0 


60 


28.6 


60 


GMAT-Total 


418.0 


31 


389.8 


23 


396.4 


26 


TOEFL Total 


589.4 


89 


541.8 


71 


552.9 


77 


TOEFL LC® 


58.2 


83 


54.8 


71 


55.6 


75 


TOEFL S & WE 


58.7 


88 


53.0 


68 


54.3 


72 


TOEFL RC & V 


59.9 


88 


54.8 


67 


56.0 


74 



a 

TOEFL sections, abbreviated here and In Tables 2 and 3, are LC: Listen- 
ing Comprehension; S & WE: Structture and Written Expression; and RC & V: 
Reading Comprehension and Vocabulary. 



The correlations between section and total scores on the GHAT and 
TOEFL were nearly identical to those reported by Powers (1980). They show 
that the TOEFL scores were strongly reiatred to The ^MAT verbal scores but 
only moderately related to the quantitative scores. The correlation 
between TOEFL and the GMAT-V was slightly higher for the EPL group (£ - 
.76) than for the ESL group (£ .68). However, the EPL group attained 
higher scores on the GMAT-V than did the ESL group. Thus, the lower 
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correlafion for the ESL group may have resulted from the fact that this 
group had lower and more homogeneous scores on the GMAT, thus producing a 
possible cellar effect. Also, because of the EPL examinees greater 
proficiency in English, their performance may have been more reliably 
assessed by the GMAT. Although it is not possible to identify Precisely 
the cause of the discrepancy in the correlations for the EPL and ESL 
groups, the differences are small enough that, for purposes of interpret- 
ing GMAT scores in light of TOEFL scores, separate treatment of EPL and 
ESL subgroups may not be necessary. 



GRE /TOEFL 

Table 2 shows the means and percentile ranks for nearly 4,000 ORE/ 
TOEFL examinees. Reference groups for deriving the percentile ranks here 
are (a) for TOEFL, all graduate-level candidates who took TOEFL between 
October 1977 and August 1980. and (b) for the GRE, all candidates took 
the GRE between October 1977 and June 1980. The data show that the 
typical GRE/TOEFL examinee was above average In English language profi- 
ciency, as reflected in TOEFL performance (80th percentile). Performance 
on the GRE, in contrast, was considerably lower for this subgroup of 
examinees than for all GRE examinees on the verbal and analytical aptitude 
sections, although it was above average for the quantitative section. 
While the foreign EPL examinees exhibited higher ranks on TOEFL than did 
their ESL counterparts (84 vs. 76). they were still substantially below 
the mean for all GRE examinees on the GRE verbal and analytical subtests. 
The foreign EPL candidates performed better than the ESL group on the 
verbal section of the GRE but not on the other two sections. 



Table 2 

Means and Percentile Ranks for TOEFL/GRE Examinees 





Foreign EPL 
(N - 1366) 


Foreign ESL 
(N - 2422) 


EPL 
( j< - 


& ESL 
3808) 




Mean 


Rank 


Mean 


Rank 


Mean 


Rank 


GRE-Verbal 

GRE-Quantitative 

GRE-Analytical 


386 
603 
406 


23 
72 
24 


345 
606 
400 


16 
72 
23 


360 
605 
402 


18 
72 
23 


TOEFL Total 
TOEFL LC 
TOEFL S & WE 
TOEFL RC & V 


573 
56 
57 
59 


84 
76 
83 
84 


552 
55 
54 
56 


76 
72 
72 
74 


559 
55 
55 
57 


80 
72 
76 
78 



ERIC 
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Table 3 depicts the relationships between GRE and TOEFL scores for 
the EPL, ESL, and combined groups of GRE/TOEFL examinees. The combined- 
group data show that TOEFL total score was strongly related to the GRE 
verbal score (£ - .70), moderately related to the GRE analytical score 
(£ - .62), and only weakly related to the GRE quantitative score (r • 
.21). The correlations between TOEFL and the GRE verbal and analytical 
measures were slightly higher for the EPL than the ESL group. Among TOEFL 
sections, Li8t.jning Comprehension consistently correlated lowest with 
the GRE scorea. This finding is not surprising since the GRE does not 
assess listening skills. Of the three TOEFL sections. Beading Comprehen- 
sion and Vocabulary showed the highest correlation with the GRE verbal 
scores, which is not surprising either, since reading is more consistently 
required for successful performance on all three sections of the GRE than 
is listening or aspects of writing tapped by the TOEFL Structure and 
Written Expression section. In general, the pattern of correlations 
supports the construct validity of the three-section TOEFL. 



Table 3 

Correlations between Scores on TOEFL and Scores on the 
GRE Aptitude Test for EPL, ESL, and Combined Group 



GRE~Verbal GRE-Quantltative GRE-Analytical 
EPL ESL CoiaH. ' t?h ESL Comb. EPL ESL Comb. 

X. 



TOEFL 


Total 


.74 


.X" 

.66 


.70 


.21 


.21 


.21 


.64 


.61 


.62 


TOEFL 


LC 


.60 


.48 


.52 


.14 


.08 


.10 


.53 


.47 


.49 


TOEFL 


S & WE 


.69 


.56 


.63 


.20 


.22 


.21 


.58 


.52 


.53 


TOEFL 


RC & V 


.72 


,67 


.70 


.23 


.25 


.24 


.61 


.58 


.59 



A general conclusion suggested by the data is that limited English 
proficiency Is a major factor contributing to the depressed scores of 
foreign examinees on the GRE and the GMAT. 
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81 Woodford, P. E. (1982). The Teat of Engllah for International 
Co»«unlc«tlon (TOEIC). In C. Brumflt (Ed.). E^^hIIbh for Inter- 
national Coammlcatlon . Oxford: Pergaroon J-reas. (ERIC Docuaent 
Reproductlo Service No. ED 198 146) 

Purpose 

This paper provides a basic overview of the Test of English for 
International Cotaiainicatlon (TOEIC). 

Method 

The TOEIC is a secure lailtiple-choice test of the ability to under- 
stand spoken and written English that is published by Educational Testing 
Service. It consists of two sections: Listening Comprehension and 
Reading Comprehension. Each section contains 100 lt«.s. Separate scaled 
scores are provide for each section. Section scores range froa 5 to 495. 
thus, total scaled scores range from 10 to 990. The first form of the 
TOEIC was administered to 2.710 Japanese adults ^JJ^' T^!" 

this population, several samples representing five different section score 
rangeV were selected. Section scores were compared with scores obtained 
on direct measures of listening comprehension, speaking, reading, and 
writing. These included a measure of the ability to answer questions 
asked m the native language by a Japanese examiner, a similar 
involving comprehension of written English, a letter-writing exercise, and 
an oral English proficiency interview. One hundred eighty-seven of the 
TOEIC examinees also took TOEFL in a special administration. 

Results and Conclusions 

The subjects were divided into five proficiency levels according to 
their section scores on the TOEIC. The mean TOEIC score of each group was 
then contrasted with the mean score on each direct measwce of language 
skills and* on the Listening Comprehension and Reading Comprehension 
sections of TOEFL. Although no correlation coefficients were computed. 
Sean scores on all the direct measures and on TOEFL Ustening Reading 
Coirehenslon showed a consistent relationship with mean TOBIC section and 
total scores, as shown in comparisons of the five groups. 
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13 



82. Yalden, J. (1978), TOEFL and the management of foreign student 
enrollments. TESL Talk, 9, 16-21. 



Purpose 



This essay considers the issue whether the TOEFL is an appropriate 
instrument for making admissions decisions for foreign applicants to 
Canadian colleges. 



Discussion 



In using TOEFL scores, two major issues must be considered: (a) how 
to Interpret the scores and (b) whether TOEFL provides appropriate 
information. Regarding score interpretation, the TOEFL manual states five 
principles for the use of TOEFL in admissions decisions: (a) consider 
part scores as well as the total score; (b) consider the different kinds 
and levels of proficiency required in different fields and levels of study 
and the colleges' resources for developing students' English language 
skills; (c) do not use "cutoff* scores; (d) use all available relevant 
information, not just TOEFL scores; and (e) assemble information on the 
validity of admissions decisions based on TOEFL scores (Educational 
Testing Service, 1973). The manual also reports score ranges found 
acceptable by institutions. 

It has been argued that Educational Testing Service (ETS) is incon- 
sistent in its instructions, stating the above-mentioned principles while, 
at the same time, offering acceptable score ranges to assist in score 
interpretation. Actually, the manual is clear in cautioning users against 
misuse of TOEFL scores; yet there is some evidence that certain cautions 
provided in the manual are being ignored. In a survey of Canadian colleges 
and universities that admit foreign sttidents for a particular program, a 
fixed score on TOEFL was mentioned as a condition for acceptance by almost 
all respondents, without reference to part scores or to use of other 
measures. 

In seeking to ensure the proper use of TOEFL scores, it is essential 
to consider the second of the two main issues mentioned above: whether 
TOEFL provides appropriate information. It has been argued that TOEFL, as 
an American-made instrument, is not appropriate for use in Canada. This 
is not a major problem, however, since Canadian English is similar enough 
to the type of English tested by TOEFL. The argument that TOEFL is an 
unreliable test is also inappropriate, as research has shown that stu- 
dents' scores do not vary greatly across adrainist rations within a specifi- 
ed time frame. 

Arguments about TOEFL's validity have also been raised. Studies at 
ETS and elsewhere have addressed several aspects of validity. Concurrent 
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validlty ha« been shown in high correlations of TOEFL with similar tests 
of English language proficiency as well as with quite different types of 
test, such as the close test. Concerning predictive validity, relatively 
low correlations have been observed between TOEFL and later grade-point 
average (CPA). ETS notes, however, that prediction of GPA is not an 
appropriate criterion for ev^uatlng TOEFL. ^ The fact that TOEFL differ- 
entiates among foreign students but not among native American students 
(Angoff & Sharon. 1971) is taken by those authors as evidence of construct 
validity, in showing that TOEFL avoids the types of discriminations for 
which it was not intended. 

It must be asked whether the TOEFL subtests measure the skills 
they are intended to measure, in light of findings such as that J>»^ell 
(1970) that the cloze test correlates more highly with the Listening 
Comprehension section of TOEFL than with any other section. To understand 
the origin of TOEFL subtests requires looking at different trends in 
language assessment. 

Three trends have been Identified. 2 During the prescientif ic 
trend. Issues of objectivity and reliability were ignored and teachers 
were assumed capable of Judging students' skill. The second, psychometric- 
structuralist trend, grew out of a school of thought in which phonology, 
morphology, syntax, and semantics were believed to cwaprlse a hierarchy of 
speech levels and that these levels should be assessed separately, a^ 
should the four skills, listening, speaking, reading, and writing. 
TOEFL was developed out of this "discrete-point" testing approach and 
contains subsections to measure separate skills. 

The third, integrative-sociollnguistic trend. Involves testing of 
overall language proficiency, according to the view that an underlying 
general language competence is common to all of the modalities of language 
use (cf., Irticles by Spolsky and Oiler cited in footnotes). In recent 
studies, -integrative" testing techniques derived from this view have been 
found to intercorrclate at high levels. 

It is not clear at this stage whether the discrete-point or integra- 
tive approach is best, and perhaps both approaches have their merits. 
What is needed is a functional definition of the levels of language 



^Educational Testing Service. (1970). TOEFL manual of Interpretive 
information . Princeton, NJ: Author. 

^Spolsky. B. (1978). Introduction: Linguists and language testers. 
In B. Spolsky (Ed.), Aonroachea to language teti^itiK> JP^per* m applied 
linguistics, advances in language te sting series; 2> (Pp. 5-9). Arling- 
ton, VA: Center for Applied Linguistics. 

^Oller, J. W., Jr. (1976). A program for language testing research. 
In H. D. Brown (Ed.). Papers in second language acquisition. Ann Arbor, 
MI: Language Learning Special Issue No. 4. 
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competence required for acadeolc study. Until such a definition can be 
reached, discrete point testing is the only available alternative. It 
la hoped that a different type of testing approach can eventually be 
developed based on functional statements for each situation In which 
proficiency is to be determined. Mearorhlle, those involved in adnisslons 
decisions should be made aware of TOEFL *s purpose and how TOEFL scores 
slu}uld be Interpreted. 
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